Re: [VOTE] Release Apache ORC 1.9.3 (RC0)
+1 - Built and tested C++/Java on macOS (intel) 14.4 and openjdk 17.0.10 - Shasum and gpg sig checked Start 1: orc-test 1/8 Test #1: orc-test . Passed6.90 sec Start 2: java-test 2/8 Test #2: java-test Passed 201.50 sec Start 3: java-tools-test 3/8 Test #3: java-tools-test .. Passed0.16 sec Start 4: java-bench-gen-test 4/8 Test #4: java-bench-gen-test .. Passed1.53 sec Start 5: java-bench-scan-test 5/8 Test #5: java-bench-scan-test . Passed1.13 sec Start 6: java-bench-hive-test 6/8 Test #6: java-bench-hive-test . Passed 59.61 sec Start 7: java-bench-spark-test 7/8 Test #7: java-bench-spark-test Passed 263.71 sec Start 8: tool-test 8/8 Test #8: tool-test Passed 11.21 sec 100% tests passed, 0 tests failed out of 8 Total Test time (real) = 545.76 sec Built target test-out Thank you Gang for leading this release. Best, Yiqun William H. 于2024年3月20日周三 05:34写道： > +1 > > Thank you Gang for leading this release. > > Best Regards, > William > > On Mon, Mar 18, 2024 at 10:51 PM Shaoyun Chen wrote: > > > +1 (non-binding) > > > > I checked the following things: > > > > - signatures are good. > > ``` > > gpg --verify orc-1.9.3.tar.gz.asc > > ``` > > - checksums are good. > > ``` > > sha256sum --check orc-1.9.3.tar.gz.sha256 > > ``` > > - build and test success from source code. > > > > Ashish 于2024年3月19日周二 09:09写道： > > > > > > +1 (non-binding) > > > > > > - validated the checksum > > > - ran the java build, all test cases pass on M1 Mac on java 1.8 > > > > > > bash-5.1$ java -version > > > openjdk version "1.8.0_342" > > > > > > On Mon, Mar 18, 2024 at 9:36 AM Dongjoon Hyun > > wrote: > > > > > > > +1 > > > > > > > > - Verified the checksum and sigs > > > > - Verified the source tarball > > > > - Built and tested from source with Java 17.0.10 on MacOS > > > > 14.4/AppleSilicon. > > > > - Verified the published Maven Artifacts are built with Java 8. > > > > > > > > $ javap -verbose Reader.class > > > > Classfile /Users/dongjoon/RC/org/apache/orc/Reader.class > > > > Last modified Jan 1, 2000; size 3158 bytes > > > > SHA-256 checksum > > > > 30159ee5943ae676d807605a87206721a8761b2defbe5c9c1e8c3860c19be0a6 > > > > Compiled from "Reader.java" > > > > public interface org.apache.orc.Reader extends java.io.Closeable > > > > minor version: 0 > > > > major version: 52 > > > > > > > > > > > > Thank you so much, Gang. > > > > > > > > Dongjoon. > > > > > > > > On 2024/03/18 04:27:50 Gang Wu wrote: > > > > > Please vote on releasing the following candidate as Apache ORC > > version > > > > > 1.9.3. > > > > > > > > > > [ ] +1 Release this package as Apache ORC 1.9.3 > > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > TAG: > > > > > https://github.com/apache/orc/releases/tag/v1.9.3-rc0 > > > > > > > > > > RELEASE FILES: > > > > > https://dist.apache.org/repos/dist/dev/orc/v1.9.3-rc0 > > > > > > > > > > STAGING REPOSITORY: > > > > > > https://repository.apache.org/content/repositories/orgapacheorc-1078 > > > > > > > > > > LIST OF ISSUES: > > > > > https://issues.apache.org/jira/projects/ORC/versions/12353961 > > > > > https://github.com/apache/orc/milestone/25?closed=1 > > > > > > > > > > This vote will be open for at least 72 hours. > > > > > > > > > > Thanks, > > > > > Gang > > > > > > > > > > > > > > > > > > -- > > > thanks > > > ashish > > >
Re: [VOTE] Release Apache ORC 2.0.0 (RC0)
+1 - Built and tested C++/Java on macOS (intel) 14.3.1 and openjdk 17.0.10 - Shasum and gpg sig checked Start 1: orc-test 1/9 Test #1: orc-test . Passed6.50 sec Start 2: java-test 2/9 Test #2: java-test Passed 202.75 sec Start 3: java-examples-test 3/9 Test #3: java-examples-test ... Passed0.72 sec Start 4: java-tools-test 4/9 Test #4: java-tools-test .. Passed0.15 sec Start 5: java-bench-gen-test 5/9 Test #5: java-bench-gen-test .. Passed1.52 sec Start 6: java-bench-scan-test 6/9 Test #6: java-bench-scan-test . Passed1.24 sec Start 7: java-bench-hive-test 7/9 Test #7: java-bench-hive-test . Passed 60.09 sec Start 8: java-bench-spark-test 8/9 Test #8: java-bench-spark-test Passed 262.93 sec Start 9: tool-test 9/9 Test #9: tool-test Passed 11.89 sec 100% tests passed, 0 tests failed out of 9 Total Test time (real) = 547.82 sec Built target test-out Thank you to all contributors of version 2.0.0 Best, Yiqun Shaoyun Chen 于2024年3月5日周二 11:32写道： > +1 (non-binding) > > I checked the following things: > > - signatures are good. > ``` > gpg --verify orc-2.0.0.tar.gz.asc > ``` > - checksums are good. > ``` > sha256sum --check orc-2.0.0.tar.gz.sha256 > ``` > - build and test success from source code. > > Dongjoon Hyun 于2024年3月5日周二 11:14写道： > > > > +1 > > > > I verified RC0 with Apache Spark 4.0.0 integration tests. > > > > In addition, you can see the full summary of various tests in the > following release issue. > > - https://github.com/apache/orc/issues/1669 > > > > Thanks, > > Dongjoon. > > > > On 2024/03/05 01:37:55 Gang Wu wrote: > > > +1 > > > > > > - Verified checksum and signature. > > > - Built and run test cases for both C++ and Java. > > > Java version: openjdk 17.0.4.1 2022-08-12 > > > Clang version: Homebrew clang version 17.0.6 > > > > > > Test project /Users/gangwu/RC/ > > > dist.apache.org/repos/dist/dev/orc/v2.0.0-rc0/orc-2.0.0/build > > > Start 1: orc-test > > > 1/9 Test #1: orc-test . Passed5.02 sec > > > Start 2: java-test > > > 2/9 Test #2: java-test Passed 118.90 sec > > > Start 3: java-examples-test > > > 3/9 Test #3: java-examples-test ... Passed0.54 sec > > > Start 4: java-tools-test > > > 4/9 Test #4: java-tools-test .. Passed0.08 sec > > > Start 5: java-bench-gen-test > > > 5/9 Test #5: java-bench-gen-test .. Passed1.28 sec > > > Start 6: java-bench-scan-test > > > 6/9 Test #6: java-bench-scan-test . Passed0.93 sec > > > Start 7: java-bench-hive-test > > > 7/9 Test #7: java-bench-hive-test . Passed 11.54 sec > > > Start 8: java-bench-spark-test > > > 8/9 Test #8: java-bench-spark-test Passed 214.67 sec > > > Start 9: tool-test > > > 9/9 Test #9: tool-test Passed9.68 sec > > > > > > 100% tests passed, 0 tests failed out of 9 > > > > > > Total Test time (real) = 362.68 sec > > > Built target test-out > > > > > > Thanks Dongjoon and everyone for making 2.0.0 happen! > > > > > > Best, > > > Gang > > > > > > On Tue, Mar 5, 2024 at 8:05 AM Dongjoon Hyun > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 2.0.0. This vote is open until March 8th 1AM (PST) and passes if a > majority > > > > +1 PMC votes are cast, with a minimum of 3 +1 votes. > > > > > > > > [ ] +1 Release this package as Apache ORC 2.0.0 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v2.0.0-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v2.0.0-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1077 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12341290 > > > > https://github.com/apache/orc/milestone/20?closed=1 > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > >
Re: [VOTE] Release Apache ORC 1.7.10 (RC0)
+1 - Built and tested C++/Java on macOS 14.1 and openjdk 17.0.9 - Shasum and gpg sig checked Thank you for driving this release, Dongjoon. Best, Yiqun mystic lama 于2023年11月7日周二 00:58写道： > +1 (non-binding) > > - verified shasum > - RAT check - no violations reported > - Java build > - Java 8 - Builds successfully, all tests pass > - Java 11 - Builds successfully, all tests pass > > On Mon, 6 Nov 2023 at 08:32, William H. wrote: > > > +1 > > > > Bests, > > William > > > > > > On Sun, Nov 5, 2023 at 10:55 PM Gang Wu wrote: > > > > > +1 > > > > > > ➜ build make test-out > > > Test project /Users/gangwu/RC/ > > > dist.apache.org/repos/dist/dev/orc/v1.7.10-rc0/orc-1.7.10/build > > > Start 1: orc-test > > > 1/8 Test #1: orc-test . Passed3.27 sec > > > Start 2: java-test > > > 2/8 Test #2: java-test Passed 111.21 sec > > > Start 3: java-tools-test > > > 3/8 Test #3: java-tools-test .. Passed0.09 sec > > > Start 4: java-bench-gen-test > > > 4/8 Test #4: java-bench-gen-test .. Passed1.65 sec > > > Start 5: java-bench-scan-test > > > 5/8 Test #5: java-bench-scan-test . Passed1.03 sec > > > Start 6: java-bench-hive-test > > > 6/8 Test #6: java-bench-hive-test . Passed 11.23 sec > > > Start 7: java-bench-spark-test > > > 7/8 Test #7: java-bench-spark-test Passed5.42 sec > > > Start 8: tool-test > > > 8/8 Test #8: tool-test Passed 10.88 sec > > > > > > 100% tests passed, 0 tests failed out of 8 > > > > > > Total Test time (real) = 144.78 sec > > > Built target test-out > > > > > > On Mon, Nov 6, 2023 at 9:52 AM Dongjoon Hyun > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.7.10. This vote is open until November 10th 1AM (PST) and passes > if a > > > > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.7.10 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.7.10-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.10-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1075 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12353285 > > > > https://github.com/apache/orc/milestone/19?closed=1 > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.8.6 (RC0)
+1 - Built and tested C++/Java on macOS 14.1 and openjdk 17.0.9 - Shasum and gpg sig checked Thank you for driving this release, Dongjoon. Bests, Yiqun mystic lama 于2023年11月7日周二 00:47写道： > +1 (non-binding) > > - validated shasum > - RAT Check - no violation reported > - Java Build > - Java 8 - Builds successfully, all tests pass > - Java11 - Builds successfully, all tests pass > > On Mon, 6 Nov 2023 at 08:31, William H. wrote: > > > +1 > > > > Thank you Dongjoon for driving this release! > > > > Bests, > > William > > > > > > On Sun, Nov 5, 2023 at 11:04 PM Gang Wu wrote: > > > > > +1 > > > > > > Built with patch: https://github.com/apache/orc/pull/1580 > > > > > > Test project /Users/gangwu/RC/ > > > dist.apache.org/repos/dist/dev/orc/v1.8.6-rc0/orc-1.8.6/build > > > Start 1: orc-test > > > 1/8 Test #1: orc-test . Passed5.16 sec > > > Start 2: java-test > > > 2/8 Test #2: java-test Passed 137.64 sec > > > Start 3: java-tools-test > > > 3/8 Test #3: java-tools-test .. Passed0.08 sec > > > Start 4: java-bench-gen-test > > > 4/8 Test #4: java-bench-gen-test .. Passed1.79 sec > > > Start 5: java-bench-scan-test > > > 5/8 Test #5: java-bench-scan-test . Passed1.01 sec > > > Start 6: java-bench-hive-test > > > 6/8 Test #6: java-bench-hive-test . Passed 11.20 sec > > > Start 7: java-bench-spark-test > > > 7/8 Test #7: java-bench-spark-test Passed6.15 sec > > > Start 8: tool-test > > > 8/8 Test #8: tool-test Passed 10.81 sec > > > > > > 100% tests passed, 0 tests failed out of 8 > > > > > > Total Test time (real) = 173.85 sec > > > Built target test-out > > > > > > On Mon, Nov 6, 2023 at 9:49 AM Dongjoon Hyun > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.8.6. This vote is open until November 10th 1AM (PST) and passes if > a > > > > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.8.6 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.8.6-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.8.6-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1074 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12353808 > > > > https://github.com/apache/orc/milestone/24?closed=1 > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.9.2 (RC0)
+1 - Built and tested C++/Java on macOS 14.1 and openjdk 17.0.9 - Shasum and gpg sig checked Thank you for driving this release, Dongjoon. Bests, Yiqun William H. 于2023年11月7日周二 00:32写道： > +1 > > Bests, > William > > > On Mon, Nov 6, 2023 at 11:11 AM mystic lama > wrote: > > > +1 (non-binding) > > > > - validated sha checksum > > - ran RAT check, no violations > > - Build > > - Java 8 - builds successfully, all test cases pass > > - Java 11- builds successfully, all test cases pass > > > > On Sun, 5 Nov 2023 at 20:50, Gang Wu wrote: > > > > > +1 > > > > > > Test project /Users/gangwu/RC/ > > > dist.apache.org/repos/dist/dev/orc/v1.9.2-rc0/orc-1.9.2/build > > > Start 1: orc-test > > > 1/8 Test #1: orc-test . Passed5.50 sec > > > Start 2: java-test > > > 2/8 Test #2: java-test Passed 120.74 sec > > > Start 3: java-tools-test > > > 3/8 Test #3: java-tools-test .. Passed0.11 sec > > > Start 4: java-bench-gen-test > > > 4/8 Test #4: java-bench-gen-test .. Passed1.72 sec > > > Start 5: java-bench-scan-test > > > 5/8 Test #5: java-bench-scan-test . Passed1.00 sec > > > Start 6: java-bench-hive-test > > > 6/8 Test #6: java-bench-hive-test . Passed 11.23 sec > > > Start 7: java-bench-spark-test > > > 7/8 Test #7: java-bench-spark-test Passed5.60 sec > > > Start 8: tool-test > > > 8/8 Test #8: tool-test Passed9.89 sec > > > > > > 100% tests passed, 0 tests failed out of 8 > > > > > > Total Test time (real) = 155.81 sec > > > Built target test-out > > > > > > On Mon, Nov 6, 2023 at 9:48 AM Dongjoon Hyun > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.9.2. This vote is open until November 10th 1AM (PST) and passes if > a > > > > majority +1 PMC votes are cast, with a minimum of 3 +1 votes. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.9.2 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.9.2-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.9.2-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1073 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12353542 > > > > https://github.com/apache/orc/milestone/23?closed=1 > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.8.5 (RC0)
Thanks Gang! +1 - Build and test C++/Java on macOS (intel) 13.5.1 with Apple clang 14.0.3 and openjdk 17.0.8.1 - Verified checksum and signature Bests, Yiqun Gang Wu 于2023年9月2日周六 22:44写道： > Please vote on releasing the following candidate as Apache ORC version > 1.8.5. > > [ ] +1 Release this package as Apache ORC 1.8.5 > [ ] -1 Do not release this package because ... > > TAG: > https://github.com/apache/orc/releases/tag/v1.8.5-rc0 > > RELEASE FILES: > https://dist.apache.org/repos/dist/dev/orc/v1.8.5-rc0 > > STAGING REPOSITORY: > https://repository.apache.org/content/repositories/orgapacheorc-1072 > > LIST OF ISSUES: > https://issues.apache.org/jira/projects/ORC/versions/12353582 > https://github.com/apache/orc/milestone/21?closed=1 > > This vote will be open for at least 72 hours (considering this email is > sent out on the weekend). > > Thanks, > Gang >
[jira] [Created] (ORC-1495) Set the assigned id for cudf
Yiqun Zhang created ORC-1495: Summary: Set the assigned id for cudf Key: ORC-1495 URL: https://issues.apache.org/jira/browse/ORC-1495 Project: ORC Issue Type: Task Reporter: Yiqun Zhang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ORC-1494) A series of tasks to add cudf writer
Yiqun Zhang created ORC-1494: Summary: A series of tasks to add cudf writer Key: ORC-1494 URL: https://issues.apache.org/jira/browse/ORC-1494 Project: ORC Issue Type: Task Components: C++, Java Reporter: Yiqun Zhang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (ORC-1490) Mistaken for a Java ORC writer when the ORC file footer does not specify a writer.
Yiqun Zhang created ORC-1490:
Summary: Mistaken for a Java ORC writer when the ORC file footer
does not specify a writer.
Key: ORC-1490
URL: https://issues.apache.org/jira/browse/ORC-1490
Project: ORC
Issue Type: Bug
Components: Java
Affects Versions: 1.9.1, 1.8.4, 1.7.10
Reporter: Yiqun Zhang
{code:java}
public OrcFile.WriterVersion getWriterVersion() {
OrcProto.PostScript ps = fileTail.getPostscript();
OrcProto.Footer footer = fileTail.getFooter();
OrcFile.WriterImplementation writer =
OrcFile.WriterImplementation.from(footer.getWriter());
return OrcFile.WriterVersion.from(writer, ps.getWriterVersion());
} {code}
We should determine footer.hasWriter() before footer.getWriter().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (ORC-1489) Assign a writer id to CUDF
Yiqun Zhang created ORC-1489: Summary: Assign a writer id to CUDF Key: ORC-1489 URL: https://issues.apache.org/jira/browse/ORC-1489 Project: ORC Issue Type: Improvement Components: C++, Java Affects Versions: 1.9.1, 1.8.4, 1.7.9 Reporter: Yiqun Zhang CUDF implements a separate orc writer that uses the gpu to write ORC files, and we should assign a writer id to it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Release Apache ORC 1.9.1 (RC0)
+1 - Built and tested C++/Java on macOS (intel) 13.4.1 and openjdk 17.0.8 - Verified checksum and signature Thanks, Yiqun Gang Wu 于2023年8月13日周日 09:46写道： > Filed two jiras: > > https://issues.apache.org/jira/browse/ORC-1475 > https://issues.apache.org/jira/browse/ORC-1476 > > On Sun, Aug 13, 2023 at 4:33 AM Dongjoon Hyun > wrote: > > > Thank you, all. > > > > For the reported cases, please file the corresponding Jira issues if they > > don’t have already. > > > > Dongjoon > > > > On Sat, Aug 12, 2023 at 10:39 William H. wrote: > > > > > +1 > > > > > > Thank you Dongjoon for driving this release. > > > > > > Bests, > > > William > > > > > > On Sat, Aug 12, 2023 at 9:18 AM mystic lama > > > wrote: > > > > > > > @Gang Wu - Thanks for the explanation. Makes sense to me. > > > > > > > > My vote +1 > > > > - verified checksum and signature > > > > - ran Java build, compilation and tests successful on M1 mac > > > > - All JIRA's look good > > > > > > > > Minor observation(s) > > > > - All PR's are in closed state, but the following 2 are in merged > > state. > > > > Just an observation > > > > - https://github.com/apache/orc/pull/1551 > > > > - https://github.com/apache/orc/pull/1552 > > > > > > > > > > > > > > > > > > > > On Sat, 12 Aug 2023 at 08:50, Gang Wu wrote: > > > > > > > > > +1 > > > > > > > > > > To Mystic: > > > > > The root cause is that the compiler uses signed char to interpret > > > > > char. The error may disappear if -funsigned-char is added to the > > > > > compiler options. I will take a look to eliminate the UB issue but > > > > > I do not think this is a blocker to the release. > > > > > > > > > > > > > > > Verified checksum and signature. > > > > > All Java and C++ build and test passed on my Mac locally. > > > > > Tool versions: openjdk 1.8.0_322 and clang 14.0.6 > > > > > > > > > > BTW, I initially failed with the following error: > > > > > [ERROR] Failed to execute goal > > > > > com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on > > project > > > > > orc-core: Error extracting protoc for version 3.17.3: Unsupported > > > > platform: > > > > > protoc-3.17.3-osx-aarch_64.exe -> [Help 1] > > > > > I did the trick from link [1] and it passed. > > > > > > > > > > [1] > > > https://github.com/os72/protoc-jar/issues/93#issuecomment-1142635897 > > > > > > > > > > Best, > > > > > Gang > > > > > > > > > > > > > > > On Sat, Aug 12, 2023 at 12:56 PM mystic lama < > > mysticlama...@gmail.com> > > > > > wrote: > > > > > > > > > > > I am getting an error when I ran docker test on my local machine > > like > > > > > > It could be a local issue, but just want to confirm before > sending > > in > > > > my > > > > > > vote > > > > > > > > > > > > Error from log towards end of email. > > > > > > > > > > > > Steps I followed > > > > > > 1. Downloaded from > > > > > https://dist.apache.org/repos/dist/dev/orc/v1.9.1-rc0/ > > > > > > 2. After extracting, ran the command from docker folder > > > > > > > > > > > > bash-5.1$ ./run-all.sh local branch-1.9 > > > > > > Launching centos7 as 85438 > > > > > > Launching debian10 as 85439 > > > > > > Launching debian11 as 85440 > > > > > > Launching ubuntu20 as 85441 > > > > > > Launching ubuntu22 as 85443 > > > > > > Launching fedora37 as 85448 > > > > > > Launching debian11_jdk=11 as 85454 > > > > > > Launching ubuntu22_jdk=11 as 85461 > > > > > > Launching ubuntu22_jdk=11_cc=clang as 85468 > > > > > > Waiting for 85438 > > > > > > Failed tests > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > (64 > > > > > ms) > > > > > > [ FAILED ] 1 test, listed below: > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > > > > 1 FAILED TEST > > > > > > FAILED centos7 > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > (19 > > > > > ms) > > > > > > [ FAILED ] 1 test, listed below: > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > > > > 1 FAILED TEST > > > > > > FAILED debian10 > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > (62 > > > > > ms) > > > > > > [ FAILED ] 1 test, listed below: > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > > > > 1 FAILED TEST > > > > > > FAILED debian11 > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > (42 > > > > > ms) > > > > > > [ FAILED ] 1 test, listed below: > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > > > > 1 FAILED TEST > > > > > > FAILED debian11_jdk=11 > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > (58 > > > > > ms) > > > > > > [ FAILED ] 1 test, listed below: > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > > > > 1 FAILED TEST > > > > > > [ FAILED ] > ConvertColumnReader.TestConvertNumericToStringVariant > > > (26 > > > > > ms) > >
Re: [VOTE] Release Apache ORC 1.9.0 (RC0)
+1 - Build and test C++/Java on macOS (intel) 13.4.1 with Apple clang 14.0.3 and openjdk 11.0.19 - Verified checksum and signature Bests, Yiqun Shaoyun Chen 于2023年6月25日周日 16:23写道： > +1 > > Both use Apple Clang version 14.0.3[1] and Clang 14.0.6[2] installed > through brew can build successfully. > > [1] https://github.com/apache/orc/pull/1551#issuecomment-1605893531 > [2] https://github.com/apache/orc/pull/1551#issuecomment-1605920181 > > Gang Wu 于2023年6月25日周日 16:14写道： > > > > +1 > > > > After I tried to enforce clang 14 via env, everything looks good now. > > If we are not dropping support of clang 13 or under, we still need > > the fix above. > > > > Best, > > Gang > > > > On Sun, Jun 25, 2023 at 12:02 PM Gang Wu wrote: > > > > > I use clang 14 on my mac m1. > > > > > > ➜ clang -v > > > Homebrew clang version 14.0.6 > > > Target: arm64-apple-darwin21.6.0 > > > Thread model: posix > > > InstalledDir: /opt/homebrew/opt/llvm@14/bin > > > > > > I think the issue can only happen on non-x86 machines. > > > This is the proposed fix: https://github.com/apache/orc/pull/1552 > > > > > > Best regards > > > Gang > > > > > > > > > > > > On Sun, Jun 25, 2023 at 11:54 AM Dongjoon Hyun < > dongjoon.h...@gmail.com> > > > wrote: > > > > > >> Thank you for testing and sharing. > > >> > > >> What is your environment? From 1.9, all docker environements are > moved to > > >> at least Clang 14+. > > >> > > >> Dongjoon > > >> > > >> On Sat, Jun 24, 2023 at 6:40 PM Gang Wu wrote: > > >> > > >> > Thanks Dongjoon! > > >> > > > >> > I have verified checksum and signature. And also can confirm that > the > > >> java > > >> > build works as expected. > > >> > > > >> > However, when I tried to run the C++ build, it complains with > following > > >> > error: > > >> > > > >> > [ 46%] Building CXX object c++/src/CMakeFiles/orc.dir/Common.cc.o > > >> > [ 47%] Building CXX object > c++/src/CMakeFiles/orc.dir/Compression.cc.o > > >> > [ 48%] Building CXX object > > >> > c++/src/CMakeFiles/orc.dir/ConvertColumnReader.cc.o > > >> > [ 48%] Building CXX object > c++/src/CMakeFiles/orc.dir/CpuInfoUtil.cc.o > > >> > [ 49%] Building CXX object > c++/src/CMakeFiles/orc.dir/Exceptions.cc.o > > >> > [ 49%] Building CXX object c++/src/CMakeFiles/orc.dir/Int128.cc.o > > >> > [ 50%] Building CXX object > > >> c++/src/CMakeFiles/orc.dir/LzoDecompressor.cc.o > > >> > /Users/gangwu/RC/ > > >> > > > >> > > > >> > dist.apache.org/repos/dist/dev/orc/v1.9.0-rc0/orc-1.9.0/c++/src/CpuInfoUtil.cc:65:9 > > >> > : > > >> > error: macro is not used [-Werror,-Wunused-macros] > > >> > #define UNUSED(x) (void)(x) > > >> > ^ > > >> > 1 error generated. > > >> > make[2]: *** [c++/src/CMakeFiles/orc.dir/CpuInfoUtil.cc.o] Error 1 > > >> > make[2]: *** Waiting for unfinished jobs > > >> > make[1]: *** [c++/src/CMakeFiles/orc.dir/all] Error 2 > > >> > make: *** [all] Error 2 > > >> > > > >> > Let me investigate and update with a fix. > > >> > > > >> > Best, > > >> > Gang > > >> > > > >> > On Sat, Jun 24, 2023 at 3:23 PM Dongjoon Hyun < > dongjoon.h...@gmail.com> > > >> > wrote: > > >> > > > >> > > I'll start with my +1. > > >> > > > > >> > > RC0 is tested with all test coverages including unit tests, > multiple > > >> > docker > > >> > > os tests, Spark and Iceberg integration tests. Here is the > summary. > > >> > > > > >> > > https://github.com/apache/orc/issues/1532 > > >> > > > > >> > > Dongjoon. > > >> > > > > >> > > > > >> > > > > >> > > On Fri, Jun 23, 2023 at 5:47 PM Dongjoon Hyun < > dongj...@apache.org> > > >> > wrote: > > >> > > > > >> > > > Please vote on releasing the following candidate as Apache ORC > > >> version > > >> > > > 1.9.0. This vote is open until June 28th 1AM (PST) and passes > if a > > >> > > majority > > >> > > > +1 PMC votes are cast, with a minimum of 3 +1 votes. > > >> > > > > > >> > > > [ ] +1 Release this package as Apache ORC 1.9.0 > > >> > > > [ ] -1 Do not release this package because ... > > >> > > > > > >> > > > TAG: > > >> > > > https://github.com/apache/orc/releases/tag/v1.9.0-rc0 > > >> > > > > > >> > > > RELEASE FILES: > > >> > > > https://dist.apache.org/repos/dist/dev/orc/v1.9.0-rc0 > > >> > > > > > >> > > > STAGING REPOSITORY: > > >> > > > > > >> https://repository.apache.org/content/repositories/orgapacheorc-1070 > > >> > > > > > >> > > > LIST OF ISSUES: > > >> > > > https://issues.apache.org/jira/projects/ORC/versions/12351711 > > >> > > > https://github.com/apache/orc/milestone/10?closed=1 > > >> > > > > > >> > > > > > >> > > > Thanks, > > >> > > > Dongjoon. > > >> > > > > > >> > > > > >> > > > >> > > > >
[ANNOUNCE] Announcing Apache ORC 1.8.4
Hi All. We are happy to announce the availability of Apache ORC 1.8.4! https://orc.apache.org/news/2023/06/14/ORC-1.8.4/ 1.8.4 is a maintenance release containing important fixes. It's available in Apache Downloads and Maven Central. https://downloads.apache.org/orc/orc-1.8.4/ https://repo1.maven.org/maven2/org/apache/orc/orc-core/1.8.4/ Cheers, Yiqun.
[VOTE][RESULT] Release Apache ORC 1.8.4 (RC0)
The vote passed. Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Gang Wu * - Deshan Xiao - Yiqun Zhang * - William Hyun * - Shaoyun Chen +0: None -1: None
Re: [VOTE] Release Apache ORC 1.8.4 (RC0)
Thank you all I'll conclude this vote. Yiqun Shaoyun Chen 于2023年6月12日周一 14:55写道： > +1 > > William H. 于2023年6月12日周一 14:50写道： > > > > +1 > > > > Thank you for driving this release! > > > > Bests, > > William > > > > On Sun, Jun 11, 2023 at 11:40 PM Yiqun Zhang > wrote: > > > > > +1 > > > > > > As the release manager for this version, here are the results of my > tests > > > - https://github.com/apache/orc/issues/1528 > > > * Checksum and signature are verified. > > > * Java 8/11/17 test passed > > > * CentOS7, Debian10/11, Ubuntu18/20/22, Fedora37 test passed > > > * Apache Spark and Apache Iceberg integration test passed. > > > > > > Thank you all. > > > > > > Yiqun > > > > > > deshan xiao 于2023年6月12日周一 10:37写道： > > > > > > > +1 > > > > > > > > Thanks for driving the release, Yiqun! > > > > > > > > Thanks, > > > > Deshan > > > > > > > > On Mon, Jun 12, 2023 at 10:08 AM Gang Wu wrote: > > > > > > > > > +1 > > > > > > > > > > Thanks Yiqun! > > > > > > > > > > Best, > > > > > Gang > > > > > > > > > > On Mon, Jun 12, 2023 at 4:19 AM Dongjoon Hyun > > > > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > Thank you so much, Yiqun. > > > > > > > > > > > > Dongjoon. > > > > > > > > > > > > On 2023/06/11 02:58:13 Yiqun Zhang wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > > > > version > > > > > > > 1.8.4. > > > > > > > > > > > > > > [ ] +1 Release this package as Apache ORC 1.8.4 > > > > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > > > > > TAG: > > > > > > > https://github.com/apache/orc/releases/tag/v1.8.4-rc0 > > > > > > > > > > > > > > RELEASE FILES: > > > > > > > https://dist.apache.org/repos/dist/dev/orc/v1.8.4-rc0 > > > > > > > > > > > > > > STAGING REPOSITORY: > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheorc-1069 > > > > > > > > > > > > > > LIST OF ISSUES: > > > > > > > https://issues.apache.org/jira/projects/ORC/versions/12353041 > > > > > > > https://github.com/apache/orc/milestone/18?closed=1 > > > > > > > > > > > > > > This vote will be open for 72 hours. > > > > > > > > > > > > > > Thanks, > > > > > > > Yiqun > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.8.4 (RC0)
+1 As the release manager for this version, here are the results of my tests - https://github.com/apache/orc/issues/1528 * Checksum and signature are verified. * Java 8/11/17 test passed * CentOS7, Debian10/11, Ubuntu18/20/22, Fedora37 test passed * Apache Spark and Apache Iceberg integration test passed. Thank you all. Yiqun deshan xiao 于2023年6月12日周一 10:37写道： > +1 > > Thanks for driving the release, Yiqun! > > Thanks, > Deshan > > On Mon, Jun 12, 2023 at 10:08 AM Gang Wu wrote: > > > +1 > > > > Thanks Yiqun! > > > > Best, > > Gang > > > > On Mon, Jun 12, 2023 at 4:19 AM Dongjoon Hyun > wrote: > > > > > +1 > > > > > > Thank you so much, Yiqun. > > > > > > Dongjoon. > > > > > > On 2023/06/11 02:58:13 Yiqun Zhang wrote: > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.8.4. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.8.4 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.8.4-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.8.4-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1069 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12353041 > > > > https://github.com/apache/orc/milestone/18?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Thanks, > > > > Yiqun > > > > > > > > > >
[VOTE] Release Apache ORC 1.8.4 (RC0)
Please vote on releasing the following candidate as Apache ORC version 1.8.4. [ ] +1 Release this package as Apache ORC 1.8.4 [ ] -1 Do not release this package because ... TAG: https://github.com/apache/orc/releases/tag/v1.8.4-rc0 RELEASE FILES: https://dist.apache.org/repos/dist/dev/orc/v1.8.4-rc0 STAGING REPOSITORY: https://repository.apache.org/content/repositories/orgapacheorc-1069 LIST OF ISSUES: https://issues.apache.org/jira/projects/ORC/versions/12353041 https://github.com/apache/orc/milestone/18?closed=1 This vote will be open for 72 hours. Thanks, Yiqun
Re: ORC 1.8.4 release?
Thank you for your reminder. I plan to start the Release Candidate (RC) voting for Apache ORC 1.8.4 tomorrow, June 11th, 2023. I have also updated the issue. Please let me know if there are any additional preparations I should make before starting the vote. I appreciate your continuous support and guidance in this process. Best regards, Yiqun Dongjoon Hyun 于2023年6月10日周六 16:02写道： > Hi, Yiqun. > > Thank you for making a release issue for sharing. > Could you update your status and RC vote start date? > > https://github.com/apache/orc/issues/1528 > (Release Apache ORC 1.8.4, Due by June 16th) > > Dongjoon. > > On 2023/05/25 05:06:40 Yiqun Zhang wrote: > > Thank you all! > > > > I have received the detailed step-by-step documentation from Dongjoon, > > and after reading it, I am now confident in the release process. > > > > Best, > > Yiqun > > > > William H. 于2023年5月25日周四 12:02写道： > > > > > Thank you Yiqun for volunteering as the release manager for this > version of > > > ORC! > > > > > > Do let us know if you need any guidance in achieving this. > > > > > > Also, it is good to see our release manager pool expanding, I’m looking > > > forward to seeing more growth in the future. > > > > > > Best Regards, > > > William > > > > > > On Sat, May 20, 2023 at 7:35 AM Gang Wu wrote: > > > > > > > Thanks Yiqun for volunteering! > > > > > > > > Yes, the information provided by Dongjoon is very exhaustive and I > have > > > > just released v1.7.9 smoothly. > > > > > > > > Please let me know if you have any questions. > > > > > > > > > > Best, > > > > Gang > > > > > > > > On Sat, May 20, 2023 at 3:39 AM Dongjoon Hyun > > > wrote: > > > > > > > > > Thank you, William and Yiqun! > > > > > > > > > > To Yiqun, I'll share some information to you via email. > > > > > (I also helped Gang with the same information last time. He may > help > > > you > > > > > too.) > > > > > > > > > > Dongjoon > > > > > > > > > > On Thu, May 18, 2023 at 11:59 PM Yiqun Zhang < > guiyanaku...@gmail.com> > > > > > wrote: > > > > > > > > > > > I'd be happy to volunteer as the release manager for this > version. > > > > > > Please be aware that this is a new role for me, and I may need > some > > > > > > guidance and assistance from the team. Thank you in advance for > > > > > > your understanding and support. > > > > > > > > > > > > Bests, > > > > > > Yiqun > > > > > > > > > > > > William H. 于2023年5月19日周五 14:24写道： > > > > > > > > > > > > > Hi All! > > > > > > > > > > > > > > ORC 1.8.4 is scheduled for release on June 15th! > > > > > > > > > > > > > > https://github.com/apache/orc/milestone/18 > > > > > > > > > > > > > > Would anyone like the volunteer as the release manager for this > > > > > version? > > > > > > > > > > > > > > Thank you, > > > > > > > William > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: ORC 1.8.4 release?
Thank you all! I have received the detailed step-by-step documentation from Dongjoon, and after reading it, I am now confident in the release process. Best, Yiqun William H. 于2023年5月25日周四 12:02写道： > Thank you Yiqun for volunteering as the release manager for this version of > ORC! > > Do let us know if you need any guidance in achieving this. > > Also, it is good to see our release manager pool expanding, I’m looking > forward to seeing more growth in the future. > > Best Regards, > William > > On Sat, May 20, 2023 at 7:35 AM Gang Wu wrote: > > > Thanks Yiqun for volunteering! > > > > Yes, the information provided by Dongjoon is very exhaustive and I have > > just released v1.7.9 smoothly. > > > > Please let me know if you have any questions. > > > > Best, > > Gang > > > > On Sat, May 20, 2023 at 3:39 AM Dongjoon Hyun > wrote: > > > > > Thank you, William and Yiqun! > > > > > > To Yiqun, I'll share some information to you via email. > > > (I also helped Gang with the same information last time. He may help > you > > > too.) > > > > > > Dongjoon > > > > > > On Thu, May 18, 2023 at 11:59 PM Yiqun Zhang > > > wrote: > > > > > > > I'd be happy to volunteer as the release manager for this version. > > > > Please be aware that this is a new role for me, and I may need some > > > > guidance and assistance from the team. Thank you in advance for > > > > your understanding and support. > > > > > > > > Bests, > > > > Yiqun > > > > > > > > William H. 于2023年5月19日周五 14:24写道： > > > > > > > > > Hi All! > > > > > > > > > > ORC 1.8.4 is scheduled for release on June 15th! > > > > > > > > > > https://github.com/apache/orc/milestone/18 > > > > > > > > > > Would anyone like the volunteer as the release manager for this > > > version? > > > > > > > > > > Thank you, > > > > > William > > > > > > > > > > > > > > >
Re: ORC 1.9 early release?
This looks excellent to me. + 1 Best, Yiqun Dongjoon Hyun 于2023年5月21日周日 15:57写道： > Thank you, Gang! > > Dongjoon > > On Sat, May 20, 2023 at 9:45 PM Wu Gang wrote: > > > > > SGTM. +1 > > > > Best, > > Gang > > > > > 在 2023年5月21日，12:28，Dongjoon Hyun 写道： > > > > > > Hi, All. > > > > > > I'd like to propose adjusting Apache ORC 1.9 release date to the end of > > June > > > because the Apache Spark community sets Apache Spark 3.5 feature freeze > > > date on July. > > > > > > https://spark.apache.org/versioning-policy.html > > >> July 16th 2023 Code freeze. Release branch cut. > > >> August 2023Release candidates (RC) > > > > > > WDTY? I'll volunteer as a release manager for Apache ORC 1.9.0. > > > If we deliver 1.9.0 before code freeze, we can support the Spark > > community > > > with subsequent releases like 1.9.1 (if needed) during the QA period. > > It's > > > the best for both communities. > > > > > > To sum up, new release plans will be the following. > > > > > > Apache ORC 1.8.4 (2023-06-06, Yiqun Zhang) => Apache Spark 3.4.1 > > > Apache ORC 1.9.0 (2023-06-30, Dongjoon Hyun) => Apache Spark 3.5.0 > > > > > > Dongjoon. > > >
Re: ORC 1.8.4 release?
I'd be happy to volunteer as the release manager for this version. Please be aware that this is a new role for me, and I may need some guidance and assistance from the team. Thank you in advance for your understanding and support. Bests, Yiqun William H. 于2023年5月19日周五 14:24写道： > Hi All! > > ORC 1.8.4 is scheduled for release on June 15th! > > https://github.com/apache/orc/milestone/18 > > Would anyone like the volunteer as the release manager for this version? > > Thank you, > William >
Re: [VOTE] Release Apache ORC 1.7.9 (RC1)
+1 Appreciate everyone's efforts! Bests, Yiqun William H. 于2023年5月6日周六 10:27写道： > +1 > > Thank you, All! > > Bests, > William > > On Fri, May 5, 2023 at 6:50 PM Gang Wu wrote: > > > +1 > > > > Thank you Panagiotis, for testing it with Apache Flink! > > > > Best, > > Gang > > > > On Sat, May 6, 2023 at 6:55 AM Dongjoon Hyun > > wrote: > > > > > It's great! Thank you, Panagiotis. > > > > > > Dongjoon. > > > > > > On Fri, May 5, 2023 at 2:10 PM Panagiotis Garefalakis < > > panga...@gmail.com> > > > wrote: > > > > > > > Thanks for driving this Gang! > > > > Spent some time testing this on flink as well and we got a green run > > > > < > > > > > > > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=48695=results > > > > > > > > > ! > > > > > > > > +1 from my side > > > > > > > > Cheers, > > > > Panagiotis > > > > > > > > On Fri, May 5, 2023 at 1:35 PM Dongjoon Hyun < > dongjoon.h...@gmail.com> > > > > wrote: > > > > > > > > > +1 for Apache ORC 1.7.9. > > > > > > > > > > Thank you, Gang. > > > > > > > > > > Dongjoon. > > > > > > > > > > > > > > > > > > > > On Wed, May 3, 2023 at 9:33 PM Gang Wu wrote: > > > > > > > > > > > Thanks for pointing that out! Just added. > > > > > > > > > > > > Best, > > > > > > Gang > > > > > > > > > > > > On Thu, May 4, 2023 at 11:33 AM Dongjoon Hyun < > > > dongjoon.h...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Thank you, Gang. > > > > > > > > > > > > > > BTW, the following doesn't include your JIRA issue, `ORC-1395: > > > > Enforce > > > > > > > license check via github action`. > > > > > > > > > > > > > > https://issues.apache.org/jira/projects/ORC/versions/12352940 > > > > > > > > > > > > > > GitHub Milestone includes all correctly. > > > > > > > > > > > > > > You need to include it by adding `1.7.9` into `Fixed Version` > > field > > > > of > > > > > > that > > > > > > > JIRA. > > > > > > > - https://issues.apache.org/jira/browse/ORC-1395 > > > > > > > > > > > > > > Dongjoon. > > > > > > > > > > > > > > > > > > > > > On Wed, May 3, 2023 at 7:22 PM Gang Wu > wrote: > > > > > > > > > > > > > > > Please vote on releasing the following candidate as Apache > ORC > > > > > version > > > > > > > > 1.7.9. > > > > > > > > > > > > > > > > [ ] +1 Release this package as Apache ORC 1.7.9 > > > > > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > > > > > > > TAG: > > > > > > > > https://github.com/apache/orc/releases/tag/v1.7.9-rc1 > > > > > > > > > > > > > > > > RELEASE FILES: > > > > > > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.9-rc1 > > > > > > > > > > > > > > > > STAGING REPOSITORY: > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheorc-1068 > > > > > > > > > > > > > > > > LIST OF ISSUES: > > > > > > > > > https://issues.apache.org/jira/projects/ORC/versions/12352940 > > > > > > > > https://github.com/apache/orc/milestone/17?closed=1 > > > > > > > > > > > > > > > > This vote will be open for at least 72 hours. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Gang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.7.9 (RC0)
+1 - Built and tested C++/Java on macOS 13.3.1 and openjdk 11.0.18 - Shasum and gpg sig checked Thank you for driving this release, Gang. Best, Yiqun Gang Wu 于2023年4月18日周二 13:49写道： > Please vote on releasing the following candidate as Apache ORC version > 1.7.9. > > [ ] +1 Release this package as Apache ORC 1.7.9 > [ ] -1 Do not release this package because ... > > TAG: > https://github.com/apache/orc/releases/tag/v1.7.9-rc0 > > RELEASE FILES: > https://dist.apache.org/repos/dist/dev/orc/v1.7.9-rc0 > > STAGING REPOSITORY: > https://repository.apache.org/content/repositories/orgapacheorc-1067 > > LIST OF ISSUES: > https://issues.apache.org/jira/projects/ORC/versions/12352940 > https://github.com/apache/orc/milestone/17?closed=1 > > This vote will be open for at least 72 hours. > > Thanks, > Gang >
Re: FYI: Apache Hive community finally succeeded to migrate to Apache ORC 1.8.3
Exciting news! Congratulations and thanks to everyone involved! Best, Yiqun Owen O'Malley 于2023年4月14日周五 13:50写道： > That is great! I’m glad that they can now take advantage of the continuing > ORC development. > > .. Owen > > > On Apr 14, 2023, at 04:40, Gang Wu wrote: > > > > Congrats and thanks to everyone involved! > > > > Best, > > Gang > > > >> On Fri, Apr 14, 2023 at 4:56 AM Panagiotis Garefalakis < > pga...@apache.org> > >> wrote: > >> > >> Amazing news indeed! > >> Congratulations to everyone involved! > >> > >> Cheers, > >> Panagiotis > >> > >> On Thu, Apr 13, 2023 at 1:32 PM Dongjoon Hyun > >> wrote: > >> > >>> This is a significant milestone for both Apache Hive and Apache ORC > >>> communities. > >>> > >>> HIVE-26809: Upgrade ORC to 1.8.3 > >>> https://issues.apache.org/jira/browse/HIVE-26809 > >>> https://github.com/apache/hive/pull/4121 > >>> > >>> Previously, the Apache Hive master branch used 1.6.9. > >>> > >>> We want to acknowledge all community members for contributing to the > >> 1.8.3 > >>> release. This release would not have been possible without you. > >>> > >>> Thanks, > >>> Dongjoon. > >>> > >> >
[jira] [Created] (ORC-1375) Cancel old running ci tasks when a pr has a new commit
Yiqun Zhang created ORC-1375: Summary: Cancel old running ci tasks when a pr has a new commit Key: ORC-1375 URL: https://issues.apache.org/jira/browse/ORC-1375 Project: ORC Issue Type: Task Components: Infra Affects Versions: 1.9.0 Reporter: Yiqun Zhang Assignee: Yiqun Zhang Cancel old running ci tasks when a pr has a new commit. The old tasks were based on outdated code which wasted resources and was pointless. Canceling tasks makes for a more efficient use of resources. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Release Apache ORC 1.7.8 (RC0)
+1 Thank you for driving this release, William. Best, Yiqun Gang Wu 于2023年1月17日周二 12:40写道： > +1 > > Thanks for driving this, William. > > Best, > Gang > > On Tue, Jan 17, 2023 at 12:17 PM Dongjoon Hyun > wrote: > > > +1 > > > > Thank you so much, William. > > > > Dongjoon. > > > > On Mon, Jan 16, 2023 at 1:47 PM William H. > wrote: > > > > > Please vote on releasing the following candidate as Apache ORC version > > > 1.7.8. > > > > > > [ ] +1 Release this package as Apache ORC 1.7.8 > > > [ ] -1 Do not release this package because ... > > > > > > TAG: > > > https://github.com/apache/orc/releases/tag/v1.7.8-rc0 > > > > > > RELEASE FILES: > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.8-rc0 > > > > > > STAGING REPOSITORY: > > > https://repository.apache.org/content/repositories/orgapacheorc-1065 > > > > > > LIST OF ISSUES: > > > https://issues.apache.org/jira/projects/ORC/versions/12352724 > > > https://github.com/apache/orc/milestone/14 > > > > > > This vote will be open for 72 hours. > > > > > > Regards, > > > William > > > > > >
Re: [VOTE] Release Apache ORC 1.8.2 (RC0)
+1 - Built and tested C++/Java on macOS (intel) 13.1 and openjdk 11.0.17 - Verified checksum and signature Thanks, Yiqun William H. 于2023年1月11日周三 10:42写道： > +1 > > Thank you for preparing this release! > > Bests, > William > > On Tue, Jan 10, 2023 at 6:04 PM Gang Wu wrote: > > > > +1 > > > > Tested it locally on my laptop to verify both Java and C++ builds and > tests > > have succeeded. > > > > Best, > > Gang > > > > On Wed, Jan 11, 2023 at 9:56 AM deshan xiao > wrote: > > > > > +1, non-binding > > > > > > Thanks, > > > Deshan Xiao > > > > > > On Tue, Jan 10, 2023 at 7:38 PM Dongjoon Hyun > > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.8.2. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.8.2 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.8.2-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.8.2-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1064 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12352628 > > > > https://github.com/apache/orc/milestone/15?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > >
Re: [VOTE] Release Apache ORC 1.8.1 (RC0)
+1 - Built and tested C++/Java on macOS 13.0.1 and openjdk 11.0.17 - Verified checksum and signature Thanks, Yiqun Shaoyun Chen 于2022年11月30日周三 21:56写道： > +1 > > William H. 于2022年11月30日周三 14:46写道： > > > > +1 > > > > Thank you for driving this release! > > > > Bests, > > William > > > > On Tue, Nov 29, 2022 at 6:19 PM Gang Wu wrote: > > > > > +1 > > > > > > Tested it locally to verify both Java and C++ builds and tests have > passed. > > > > > > Best, > > > Gang > > > > > > > > > On Wed, Nov 30, 2022 at 8:31 AM Dongjoon Hyun > wrote: > > > > > > > I'll start with my +1. > > > > > > > > As the release manager, I checked the following. > > > > > > > > https://github.com/apache/orc/issues/1319 (Release Apache ORC 1.8.1) > > > > > > > > - Checked shasum and signature > > > > - GitHub Action CI Check > > > > - Run docker tests > > > > - Integration tests with Apache Spark 3.4.0-SNAPSHOT > > > > - Integration tests with Apache Iceberg 1.1.0-SNAPSHOT > > > > > > > > Dongjoon > > > > > > > > On 2022/11/29 17:25:18 Dongjoon Hyun wrote: > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > > 1.8.1. > > > > > > > > > > [ ] +1 Release this package as Apache ORC 1.8.1 > > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > TAG: > > > > > https://github.com/apache/orc/releases/tag/v1.8.1-rc0 > > > > > > > > > > RELEASE FILES: > > > > > https://dist.apache.org/repos/dist/dev/orc/v1.8.1-rc0 > > > > > > > > > > STAGING REPOSITORY: > > > > > > https://repository.apache.org/content/repositories/orgapacheorc-1063 > > > > > > > > > > LIST OF ISSUES: > > > > > https://issues.apache.org/jira/projects/ORC/versions/12352296 > > > > > https://github.com/apache/orc/milestone/13?closed=1 > > > > > > > > > > This vote will be open for 72 hours. > > > > > > > > > > Thanks, > > > > > Dongjoon. > > > > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.7.7 (RC0)
+1 LGTM Thanks a lot ! Best, Yiqun Gang Wu 于2022年11月15日周二 23:22写道： > +1 > > Thanks Dongjoon for driving this! > > Best, > Gang > > On Tue, Nov 15, 2022 at 12:40 PM William H. > wrote: > > > +1 > > > > Looks good to me, thank you! > > > > On Mon, Nov 14, 2022 at 12:31 AM Dongjoon Hyun > > wrote: > > > > > I'll start my +1. > > > > > > During the preparation, I ran the Docker tests and integration tests > with > > > Apache Spark 3.2.3 (snapshot) and Apache Iceberg (1.0.x). I summarized > > the > > > result here as the release manager, > > > https://github.com/apache/orc/issues/1313 . > > > > > > Dongjoon > > > > > > On 2022/11/14 08:21:26 Dongjoon Hyun wrote: > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.7.7. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.7.7 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.7.7-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.7-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1062 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12352219 > > > > https://github.com/apache/orc/milestone/12?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > > > >
[jira] [Created] (ORC-1309) Eliminate the redundant `{}` in the Case syntax
Yiqun Zhang created ORC-1309:
Summary: Eliminate the redundant `{}` in the Case syntax
Key: ORC-1309
URL: https://issues.apache.org/jira/browse/ORC-1309
Project: ORC
Issue Type: Improvement
Components: Java
Affects Versions: 1.9.0
Reporter: Yiqun Zhang
[ORC-1306|https://issues.apache.org/jira/projects/ORC/issues/ORC-1306] caused
this issue, which was fixed by the patch to 1.8, so the main branch needs to be
fixed separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (ORC-1306) Fixed indented code style for Java modules
Yiqun Zhang created ORC-1306:
Summary: Fixed indented code style for Java modules
Key: ORC-1306
URL: https://issues.apache.org/jira/browse/ORC-1306
Project: ORC
Issue Type: Improvement
Components: Java
Affects Versions: 1.9.0
Reporter: Yiqun Zhang
Assignee: Yiqun Zhang
The ORC Java module indentation code style is not consistent, I will commit the
indentation rule and make all existing code follow it.
{code:java}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] Release Apache ORC 1.8.0 (RC0)
+1 - Built and tested C++/Java on macOS 12.5.1 and openjdk 11.0.16 - Shasum and gpg sig checked Thanks, Yiqun William H. 于2022年8月27日周六 11:40写道： > Please vote on releasing the following candidate as Apache ORC version > 1.8.0. > > [ ] +1 Release this package as Apache ORC 1.8.0 > [ ] -1 Do not release this package because ... > > TAG: > https://github.com/apache/orc/releases/tag/v1.8.0-rc0 > > RELEASE FILES: > https://dist.apache.org/repos/dist/dev/orc/v1.8.0-rc0 > > STAGING REPOSITORY: > https://repository.apache.org/content/repositories/orgapacheorc-1061/ > > LIST OF ISSUES: > https://issues.apache.org/jira/projects/ORC/versions/12350356 > https://github.com/apache/orc/milestone/2?closed=1 > > This vote will be open for 72 hours. > > Regards, > William >
Re: [VOTE] Release Apache ORC 1.7.6 (RC0)
+1 - Built and tested C++/Java on macOS 12.5 and openjdk 11.0.16 - Shasum and gpg sig checked Thanks, Yiqun Gang Wu 于2022年8月15日周一 15:47写道： > +1 > > - Built and tested C++/Java on macOS aarch64 with OpenJDK 1.8.0_322. > - Verified checksum and signature. > > Thanks, > Gang > > > On Mon, Aug 15, 2022 at 2:16 PM Dongjoon Hyun > wrote: > > > +1 > > > > Dongjoon > > > > On Sun, Aug 14, 2022 at 10:22 PM William Hyun > wrote: > > > > > I will start with my +1 > > > > > > https://github.com/apache/orc/issues/1191 > > > > > > - branch-1.7 is healthy in GitHub Action > > > - https://github.com/apache/orc/commits/branch-1.7 > > > - `Ubuntu 20.04`, `Ubuntu 22.04`, `MacOS 11.6`, `MacOS 12.5`, > `Windows` > > > are tested. > > > - `Java 8`, `Java 11`, `Java 17`, `Java 18` are tested. > > > - `Clang 11.0.0` and `g++` > > > - Docker tests (`CentOS 7`, `Debian 10`, `Debian 11`, `Ubuntu 18`, > > > `Ubuntu 20`, `Ubuntu 22`) passed. > > > - Apache Spark master integration test passed. > > > - Apache Iceberg master integration test passed. > > > > > > William > > > > > > > > > On 2022/08/15 04:57:18 "William H." wrote: > > > > Please vote on releasing the following candidate as Apache ORC > version > > > 1.7.6. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.7.6 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.7.6-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.6-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1059 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12351849 > > > > https://github.com/apache/orc/milestone/11?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Regards, > > > > William > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.7.5 (RC0)
+1 - Built and tested C++/Java on macOS 12.3.1 and openjdk 11.0.12 - Shasum and gpg sig checked Thanks, Yiqun Dongjoon Hyun 于2022年6月13日周一 07:05写道： > +1 > > Thank you, William. > > Dongjoon. > > > On Sun, Jun 12, 2022 at 2:54 PM William H. wrote: > > > I will start with my +1 > > > > https://github.com/apache/orc/issues/1155 > > > > - branch-1.7 is healthy in GitHub Action > > - Docker tests passed. > > - Apache Spark master integration test passed. > > - Apache Iceberg master integration test passed. > > > > > > > > On Sun, Jun 12, 2022 at 2:50 PM William H. > wrote: > > > > > Please vote on releasing the following candidate as Apache ORC version > > > 1.7.5. > > > > > > [ ] +1 Release this package as Apache ORC 1.7.5 > > > [ ] -1 Do not release this package because ... > > > > > > TAG: > > > https://github.com/apache/orc/releases/tag/v1.7.5-rc0 > > > > > > RELEASE FILES: > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.5-rc0 > > > > > > STAGING REPOSITORY: > > > https://repository.apache.org/content/repositories/orgapacheorc-1058 > > > > > > LIST OF ISSUES: > > > https://issues.apache.org/jira/projects/ORC/versions/12351651 > > > https://github.com/apache/orc/milestone/9?closed=1 > > > > > > This vote will be open for 72 hours. > > > > > > Regards, > > > William > > > > > >
Re: [VOTE] Release Apache ORC 1.6.14 (RC0)
+1 - Built and tested C++/Java on macOS 12.3.1 and openjdk 11.0.12 - Shasum and gpg sig checked Thanks, Yiqun William Hyun 于2022年4月13日周三 09:58写道： > +1 > > I verified checksum and gpg signature. > I also built and tested on Java 1.8 and MacOS 12.0.1. > > Thanks, > William > > On Tue, Apr 12, 2022 at 4:10 PM Quanlong Huang > wrote: > > > +1 > > > > - Built and tested C++/Java on Ubuntu 16.04 with OpenJDK 1.8.0_275. > > - Verified checksum and signature > > > > Thanks, > > Quanlong > > > > > > On Mon, Apr 11, 2022 at 12:13 PM Dongjoon Hyun > > wrote: > > > > > > I'll start with my +1. > > > > > > All test results were tracked here during preparing RC0. > > > - https://github.com/apache/orc/issues/1081 > > > > > > In addition, I also verified the uploaded artifacts on Apple Silicon > with > > > MacOS 12.3.1/Java 8. > > > > > > Dongjoon. > > > > > > > > > On Sun, Apr 10, 2022 at 8:41 PM Dongjoon Hyun > > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.6.14. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.6.14 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.6.14-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.6.14-rc0 > > > > > > > > STAGING REPOSITORY: > > > > > https://repository.apache.org/content/repositories/orgapacheorc-1057/ > > > > > > > > LIST OF JIRA ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12351429 > > > > https://github.com/apache/orc/milestone/6?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Dongjoon. > > > > > > >
Re: [VOTE] Release Apache ORC 1.7.4 (RC0)
+1 - Built and tested C++/Java on macOS 12.3.1 and openjdk 11.0.12 - Shasum and gpg sig checked Best, Yiqun Dongjoon Hyun 于2022年4月10日周日 14:23写道： > +1 > > Thank you for all your effort to prepare this release, William. > I saw your updated checklist and you did finish most testing already. > > https://github.com/apache/orc/issues/1046 > > I did the following additionally from my side > because I had an Apple Silicon notebook. > > 1. Download KEYS to import your GPG key. > 2. Shasum and GPG sig checked. > 3. Download and tested C++/Java 17 on Apple Silicon / MacOS 12.3.1. > > $ curl -LO https://dist.apache.org/repos/dist/release/orc/KEYS > $ gpg --import KEYS > $ curl -LO > https://dist.apache.org/repos/dist/dev/orc/v1.7.4-rc0/orc-1.7.4.tar.gz > $ curl -LO > https://dist.apache.org/repos/dist/dev/orc/v1.7.4-rc0/orc-1.7.4.tar.gz.asc > $ curl -LO > > https://dist.apache.org/repos/dist/dev/orc/v1.7.4-rc0/orc-1.7.4.tar.gz.sha256 > $ shasum -a256 orc-1.7.4.tar.gz | diff - orc-1.7.4.tar.gz.sha256 > $ gpg --verify orc-1.7.4.tar.gz.asc > gpg: assuming signed data in 'orc-1.7.4.tar.gz' > gpg: Signature made Sat Apr 9 15:06:22 2022 PDT > gpg:using RSA key AA94E2A8F0A0B7167305C5232D9F6201DECDFA29 > gpg: Good signature from "William Hyun (CODE SIGNING KEY) < > will...@apache.org>" [unknown] > gpg: WARNING: This key is not certified with a trusted signature! > gpg: There is no indication that the signature belongs to the > owner. > Primary key fingerprint: AA94 E2A8 F0A0 B716 7305 C523 2D9F 6201 DECD FA29 > > Dongjoon > > PS. I didn't trigger the integration tests on the staging artifacts > because I guess you are running them already. > > > > On Sat, Apr 9, 2022 at 3:48 PM William Hyun > wrote: > > > Please vote on releasing the following candidate as Apache ORC version > > 1.7.4. > > > > [ ] +1 Release this package as Apache ORC 1.7.4 > > [ ] -1 Do not release this package because ... > > > > TAG: > > https://github.com/apache/orc/releases/tag/v1.7.4-rc0 > > > > RELEASE FILES: > > https://dist.apache.org/repos/dist/dev/orc/v1.7.4-rc0 > > > > STAGING REPOSITORY: > > https://repository.apache.org/content/repositories/orgapacheorc-1056 > > > > LIST OF ISSUES: > > https://issues.apache.org/jira/projects/ORC/versions/12351349 > > https://github.com/apache/orc/milestone/7?closed=1 > > > > This vote will be open for 72 hours. > > > > Regards, > > William > > >
[jira] [Created] (ORC-1147) Use `Objects.equals(dstas.getSum(), Double.NaN)` instead of `isFinite` to determine if there is a NaN write
Yiqun Zhang created ORC-1147:
Summary: Use `Objects.equals(dstas.getSum(), Double.NaN)` instead
of `isFinite` to determine if there is a NaN write
Key: ORC-1147
URL: https://issues.apache.org/jira/browse/ORC-1147
Project: ORC
Issue Type: Improvement
Components: Java
Affects Versions: 1.6.13, 1.7.3
Reporter: Yiqun Zhang
{code:java}
if (!Double.isFinite(dstas.getSum())) {
LOG.debug("Not using predication pushdown on {} because stats contain
NaN values",
predicate.getColumnName());
return dstas.hasNull() ? TruthValue.YES_NO_NULL : TruthValue.YES_NO;
}
}
{code}
The isFinite function does not determine whether there is a NaN write or not,
and there is a possibility of a sum overflow; we should use
`Objects.equals(dstas.getSum(), Double.NaN)` instead to determine this
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (ORC-1146) Float category does not check if the statistic sum is a finite value
Yiqun Zhang created ORC-1146:
Summary: Float category does not check if the statistic sum is a
finite value
Key: ORC-1146
URL: https://issues.apache.org/jira/browse/ORC-1146
Project: ORC
Issue Type: Bug
Components: Java
Affects Versions: 1.7.3
Reporter: Yiqun Zhang
[https://github.com/apache/orc/blob/70c504cbd759342476d2fb2c9cc75a5025cb7c11/java/core/src/java/org/apache/orc/impl/RecordReaderImpl.java#L696-L703]
{code:java}
} else if (category == TypeDescription.Category.DOUBLE) {
DoubleColumnStatistics dstas = (DoubleColumnStatistics) cs;
if (!Double.isFinite(dstas.getSum())) {
LOG.debug("Not using predication pushdown on {} because stats contain NaN
values",
predicate.getColumnName());
return dstas.hasNull() ? TruthValue.YES_NO_NULL : TruthValue.YES_NO;
}
}
{code}
Currently only the Double category is checked. the Float category is also only
pushed down if the statistics sum to a finite value.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (ORC-1123) Add `estimationMemory` method for writer
Yiqun Zhang created ORC-1123: Summary: Add `estimationMemory` method for writer Key: ORC-1123 URL: https://issues.apache.org/jira/browse/ORC-1123 Project: ORC Issue Type: Improvement Components: Java Affects Versions: 1.6.13 Reporter: Yiqun Zhang Add `estimationMemory` method for writer so that the write engine can better control the ORC refresh policy. The source of this issue comes from Iceberg's discussion. https://lists.apache.org/thread/g6yo7m46mr86ov1vkm9wnmshgw7hcl6b -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release Apache ORC 1.7.3 (RC0)
+1 - C++ build and test on Mac - Java build and test on Mac env with JDK11 - Checksum and signature checked Best, Yiqun Gang Wu 于2022年2月8日周二 22:30写道： > +1. > > - Verified checksum and GPG. > - Built and ran unit tests for both Java and C++. > > Best, > Gang > > On Tue, Feb 8, 2022 at 4:00 PM William Hyun > wrote: > > > +1 > > > > Thank you for rolling the RC, it looks good to me. > > I would like to volunteer as the next release manager for 1.7.4. > > > > Cheers, > > William > > > > > > > > On Sun, Feb 6, 2022 at 11:45 PM Dongjoon Hyun > > wrote: > > > > > I'll start with my +1. > > > > > > 1. Shasum and gpg sig checked. > > > 2. Built and tested C++/Java on Apple Silicon. > > > 3. Docker tests passed (CentOS 7, Debian 9~11, Ubuntu 18, Ubuntu 20). > > > 4. Apache Spark integration test passed > > > (RC0 staging + Spark 3.3.0-SNAPSHOT) > > > https://github.com/dongjoon-hyun/spark/pull/80 > > > > > > Dongjoon. > > > > > > > > > > > > On Sun, Feb 6, 2022 at 10:05 PM Dongjoon Hyun > > > > wrote: > > > > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.7.3. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.7.3 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/v1.7.3-rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/v1.7.3-rc0 > > > > > > > > STAGING REPOSITORY: > > > > https://repository.apache.org/content/repositories/orgapacheorc-1055 > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12351162 > > > > https://github.com/apache/orc/milestone/4?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > > > >
Re: [VOTE] Release Apache ORC 1.6.13 (RC0)
+1 - C++ build and test on Mac - Java build and test on Mac env with JDK11 - Checksum and signature checked Cheers, Yiqun Dongjoon Hyun 于2022年1月18日周二 12:38写道： > I'll start with my +1. > > RC1 is tested like the following. > > - Checksum and signature checked. > - All C++/Java build and UT passed. > - Docker test passed like Apache ORC 1.6.12 > (ubuntu18, ubuntu20, centos7, centos8, debian9, debian10) > - Apache Spark Integration Test passed. > https://github.com/dongjoon-hyun/spark/pull/79 (with Apache Spark > branch-3.2) > > Please note the following. > > 1. Like the other ORC 1.6.x and older versions, > there is a known C++ compiler issue on Debian 11, ORC-1041. > 2. Unfortunately, we missed the timing for Apache Spark 3.2.1 RC2. > This might be a part of Apache Spark 3.2.2. > > Thanks, > Dongjoon. > > On 2022/01/18 03:03:08 Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache ORC version > > 1.6.13. > > > > [ ] +1 Release this package as Apache ORC 1.6.13 > > [ ] -1 Do not release this package because ... > > > > TAG: > > https://github.com/apache/orc/releases/tag/release-1.6.13rc0 > > > > RELEASE FILES: > > https://dist.apache.org/repos/dist/dev/orc/orc-1.6.13-rc0 > > > > STAGING REPOSITORY: > > https://repository.apache.org/content/repositories/orgapacheorc-1054/ > > > > LIST OF JIRA ISSUES: > > https://issues.apache.org/jira/projects/ORC/versions/12351163 > > https://github.com/apache/orc/milestone/5?closed=1 > > > > This vote will be open for 72 hours. > > > > Thanks, > > Dongjoon. > > >
[jira] [Created] (ORC-1082) FileDump and JsonFileDump do not correctly handle the case where the RowIndex does not have column statistics
Yiqun Zhang created ORC-1082:
Summary: FileDump and JsonFileDump do not correctly handle the
case where the RowIndex does not have column statistics
Key: ORC-1082
URL: https://issues.apache.org/jira/browse/ORC-1082
Project: ORC
Issue Type: Bug
Components: Java, tools
Affects Versions: 1.7.2, 1.6.12
Reporter: Yiqun Zhang
Before we can get the ColumnStatistics from RowIndex we need to determine
whether it exists or not.
entry.getStatistics(); does not return null at any time, and will return a
default object when it is not set.
{code:java}
OrcProto.ColumnStatistics colStats = entry.getStatistics();
if (colStats == null) {
buf.append("no stats at ");
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Re: [VOTE] Release Apache ORC 1.7.2 (RC0)
+1 - C++ build and test on Mac - Java build and test on Mac env with JDK11 Thank you, Yiqun Panos Garefalakis 于2021年12月18日周六 14:42写道： > Thanks for moving this forward! > > +1 from my side > > Built source code and ran Java tests > Validated checksum and GPG > > Cheers, > Panagiotis > > On Sat, Dec 18, 2021 at 2:33 AM William Hyun > wrote: > > > +1 > > > > I checked shasum and the gpg signature. > > I built and tested Java and C++ on MacOS with Java 8. > > In addition, I also tested orc-tools' new features like 'sizes'. > > > > Thank you, > > William > > > > > > On Thu, Dec 16, 2021 at 10:49 PM Dongjoon Hyun > > wrote: > > > > > +1 > > > > > > 1. Shasum and gpg sig checked. > > > 2. Docker tests passed. > > > 3. Apache Spark integration test passed (with Spark 3.3.0-SNAPSHOT) > > > > > > Dongjoon > > > > > > On 2021/12/17 06:44:00 Dongjoon Hyun wrote: > > > > Please vote on releasing the following candidate as Apache ORC > version > > > > 1.7.2. > > > > > > > > [ ] +1 Release this package as Apache ORC 1.7.2 > > > > [ ] -1 Do not release this package because ... > > > > > > > > TAG: > > > > https://github.com/apache/orc/releases/tag/release-1.7.2rc0 > > > > > > > > RELEASE FILES: > > > > https://dist.apache.org/repos/dist/dev/orc/orc-1.7.2-rc0 > > > > > > > > STAGING REPOSITORY: > > > > > https://repository.apache.org/content/repositories/orgapacheorc-1053/ > > > > > > > > LIST OF ISSUES: > > > > https://issues.apache.org/jira/projects/ORC/versions/12350842 > > > > https://github.com/apache/orc/milestone/3?closed=1 > > > > > > > > This vote will be open for 72 hours. > > > > > > > > Thanks, > > > > Dongjoon. > > > > > > > > > >
[jira] [Created] (ORC-1055) [C++] Timestamp values read in Hive are different when using ORC file created using CSV to ORC converter tools
Yiqun Zhang created ORC-1055: Summary: [C++] Timestamp values read in Hive are different when using ORC file created using CSV to ORC converter tools Key: ORC-1055 URL: https://issues.apache.org/jira/browse/ORC-1055 Project: ORC Issue Type: Bug Components: C++ Reporter: Yiqun Zhang Attachments: converted_by_cpp.orc, timestamp.csv I have a CSV file that has a column having timestamp values as 0001-01-01 00:00:00.0. Then I convert CSV file to ORC file using CSV to ORC converter and place the ORC file in a hive table backed by ORC files. On querying the data using Hive beeline and Spark SQL, different results are obtained If converted using CPP tool, value read using Hive beeline and Spark SQL queries is 0001-01-03 00:00:00 -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Release Apache ORC 1.7.1 (RC0)
+ 1 - C++ build and test on Mac - Java build and test on Mac env with JDK11 Test project /Users/yiqunzhang/Downloads/orc-release-1.7.1rc0/build Start 1: orc-test 1/6 Test #1: orc-test . Passed4.93 sec Start 2: java-test 2/6 Test #2: java-test Passed 162.58 sec Start 3: java-tools-test 3/6 Test #3: java-tools-test .. Passed0.17 sec Start 4: java-bench-gen-test 4/6 Test #4: java-bench-gen-test .. Passed1.90 sec Start 5: java-bench-scan-test 5/6 Test #5: java-bench-scan-test . Passed1.31 sec Start 6: tool-test 6/6 Test #6: tool-test Passed 14.33 sec 100% tests passed, 0 tests failed out of 6 Total Test time (real) = 185.23 sec Built target test-out Yiqun Zhang Dongjoon Hyun 于2021年11月4日周四 下午11:43写道： > +1 > > 1. Shasum and gpg sig checked. > 2. Docker tests passed except Debian 11 with a known issue,ORC-1041. > 3. Apache Spark integration test passed >(RC0 staging + Spark 3.3.0-SNAPSHOT) >https://github.com/dongjoon-hyun/spark/pull/73 > > Dongjoon > > On 2021/11/03 19:11:55 Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache ORC version > > 1.7.1. > > > > [ ] +1 Release this package as Apache ORC 1.7.1 > > [ ] -1 Do not release this package because ... > > > > TAG: > > https://github.com/apache/orc/releases/tag/release-1.7.1rc0 > > > > RELEASE FILES: > > https://dist.apache.org/repos/dist/dev/orc/orc-1.7.1-rc0 > > > > STAGING REPOSITORY: > > https://repository.apache.org/content/repositories/orgapacheorc-1051/ > > > > LIST OF ISSUES: > > https://issues.apache.org/jira/projects/ORC/versions/12350619 > > https://github.com/apache/orc/milestone/1?closed=1 > > > > This vote will be open for 72 hours. > > > > Thanks, > > Dongjoon. > > >
Re: [VOTE] Release Apache ORC 1.6.12 (RC0)
+ 1 - C++ build and test on Mac - Java build and test on Mac env with JDK11 Test project /Users/yiqunzhang/Downloads/orc-release-1.6.12rc0/build Start 1: orc-test 1/7 Test #1: orc-test . Passed3.87 sec Start 2: java-test 2/7 Test #2: java-test Passed 67.72 sec Start 3: java-examples-test 3/7 Test #3: java-examples-test ... Passed0.64 sec Start 4: java-tools-test 4/7 Test #4: java-tools-test .. Passed0.15 sec Start 5: java-bench-gen-test 5/7 Test #5: java-bench-gen-test .. Passed2.06 sec Start 6: java-bench-scan-test 6/7 Test #6: java-bench-scan-test . Passed1.34 sec Start 7: tool-test 7/7 Test #7: tool-test Passed 12.33 sec 100% tests passed, 0 tests failed out of 7 Total Test time (real) = 88.12 sec Built target test-out Yiqun Zhang Dongjoon Hyun 于2021年11月4日周四 下午11:46写道： > +1 > > 1. Shasum and gpg sig checked. > 2. Docker tests passed except Debian 11 with a known issue, ORC-1041. > 3. Apache Spark integration test passed >(RC0 staging + Spark 3.2.1-SNAPSHOT) >https://github.com/dongjoon-hyun/spark/pull/74 > > Dongjoon > > On 2021/11/03 21:11:35 Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache ORC version > > 1.6.12. > > > > [ ] +1 Release this package as Apache ORC 1.6.12 > > [ ] -1 Do not release this package because ... > > > > TAG: > > https://github.com/apache/orc/releases/tag/release-1.6.12rc0 > > > > RELEASE FILES: > > https://dist.apache.org/repos/dist/dev/orc/orc-1.6.12-rc0 > > > > STAGING REPOSITORY: > > https://repository.apache.org/content/repositories/orgapacheorc-1052/ > > > > LIST OF JIRA ISSUES: > > https://issues.apache.org/jira/projects/ORC/versions/12350643 > > > > This vote will be open for 72 hours. > > > > Thanks, > > Dongjoon. > > >
[jira] [Created] (ORC-1046) Control the parallelism of multiple OS tests
Yiqun Zhang created ORC-1046: Summary: Control the parallelism of multiple OS tests Key: ORC-1046 URL: https://issues.apache.org/jira/browse/ORC-1046 Project: ORC Issue Type: Test Components: build Reporter: Yiqun Zhang orc/docker/run-all.sh will test all OS in parallel. On my machine it's hard to run all of them successfully, some of the containers always exit with 137 error code, probably due to lack of memory, maybe we can add a parameter to control the parallelism of the script. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-1039) For the tests provided by ORC-1030 to work properly, compatibility modifications need to be provided
Yiqun Zhang created ORC-1039:
Summary: For the tests provided by ORC-1030 to work properly,
compatibility modifications need to be provided
Key: ORC-1039
URL: https://issues.apache.org/jira/browse/ORC-1039
Project: ORC
Issue Type: Test
Components: Java, tools
Affects Versions: 1.6.11, 1.7.0, 1.8.0
Reporter: Yiqun Zhang
{code:java}
Path sideFilePath = OrcAcidUtils.getSideFile(corruptPath);
Path backupSideFilePath = new Path(backupDataPath.getParent(),
sideFilePath.getName());
moveFiles(fs, sideFilePath, backupSideFilePath);
{code}
Check if the sideFile file exists. Compatibility test case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-1036) Due to tzdata upgrade, the fixed download links in CI are often not working
Yiqun Zhang created ORC-1036: Summary: Due to tzdata upgrade, the fixed download links in CI are often not working Key: ORC-1036 URL: https://issues.apache.org/jira/browse/ORC-1036 Project: ORC Issue Type: Improvement Components: build, C++ Affects Versions: 1.8.0 Reporter: Yiqun Zhang https://ci.appveyor.com/project/ApacheSoftwareFoundation/orc/builds/41273644#L152 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-1035) `backupDataPath` may be incorrect in recoverFile
Yiqun Zhang created ORC-1035:
Summary: `backupDataPath` may be incorrect in recoverFile
Key: ORC-1035
URL: https://issues.apache.org/jira/browse/ORC-1035
Project: ORC
Issue Type: Bug
Components: Java, tools
Affects Versions: 1.6.11, 1.7.0, 1.8.0
Reporter: Yiqun Zhang
{code:java}
public static final String DEFAULT_BACKUP_PATH =
System.getProperty("java.io.tmpdir");
..
String scheme = corruptPath.toUri().getScheme();
String authority = corruptPath.toUri().getAuthority();
String filePath = corruptPath.toUri().getPath();
if (backup.equals(DEFAULT_BACKUP_PATH)) {
backupDataPath = new Path(scheme, authority, DEFAULT_BACKUP_PATH +
filePath);
} else {
backupDataPath = Path.mergePaths(new Path(backup), corruptPath);
}
{code}
1. System.getProperty("java.io.tmpdir") gets a path that may or may not end in
Path.SEPARATOR, as seen in the [Travis
ci|https://app.travis-ci.com/github/apache/orc/jobs/544381649#L1580] example,
the path generated directly by DEFAULT_BACKUP_PATH + filePath may not be
correct.
2. corruptPath is the path entered by the user, which may be absolute or
relative. The second argument of Path.mergePaths is expected to be a path
starting with Path.SEPARATOR, so when the user enters a relative path,
Path.mergePaths(new Path(backup), corruptPath) the result is also incorrect.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Apache ORC Release Cadence
Thanks Dongjoon for initiating this discussion. I think the current release cadence is great, 1-2 years for a major release and we maintain each major release for about 3 years so we can make sure we maintain a consistent number of releases. https://github.com/apache/orc/pull/936#issuecomment-944017204 My opinion in response to this discussion is that we could add preview releases, such as 1.8.0-alpha, which could be released at a cadence that could be adjusted as incompatible new features accumulate. To avoid new interfaces taking too long to come out, it is also convenient for users to understand and test in advance, and we can give lower priority to fixing its issues than the major version. Regards, Yiqun Zhang Dongjoon Hyun 于2021年10月17日周日 上午11:57写道： > Hi, All. > > There were some questions and discussion about Apache ORC release cadence > at our GitHub PR. I want to share it with you and get your opinion on it. > > https://github.com/apache/orc/pull/936#issuecomment-944017204 > > According to our release page, Apache ORC release cadence requires > one or two years for the next minor version. > > https://orc.apache.org/docs/releases.html > - 1.4.0: 2017-05-08 > - 1.5.0: 2018-05-14 (About 1 year and 6 days) > - 1.6.0: 2019-09-03 (About 1 year and 4 month) > - 1.7.0: 2021-09-15 (About 2 years and 12 days) > - 1.8.0: 2023-09-15 (About 2 years?) > > Apache ORC release cadence is also related to the EOL of the old releases. > For example, > - 1.5.x was maintained for 40 months and was archived on 2021-09-15. > - 1.6.x EOL is expected on 2022-09-03 if we maintain it for 3 years. > - 1.7.x EOL is expected on 2024-09-15 if we maintain it for 3 years. > > What is your expectation on Apache ORC release cadence? > > Dongjoon >
[jira] [Created] (ORC-1034) The search byte array algorithm is incorrectly implemented in FileDump.java
Yiqun Zhang created ORC-1034:
Summary: The search byte array algorithm is incorrectly
implemented in FileDump.java
Key: ORC-1034
URL: https://issues.apache.org/jira/browse/ORC-1034
Project: ORC
Issue Type: Bug
Components: Java, tools
Affects Versions: 1.6.11, 1.7.0, 1.8.0
Reporter: Yiqun Zhang
{code:java}
private static int indexOf(final byte[] data, final byte[] pattern, final int
index) {
if (data == null || data.length == 0 || pattern == null || pattern.length
== 0 ||
index > data.length || index < 0) {
return -1;
}
int j = 0;
for (int i = index; i < data.length; i++) {
if (pattern[j] == data[i]) {
j++;
} else {
j = 0;
}
if (j == pattern.length) {
return i - pattern.length + 1;
}
}
return -1;
}
{code}
This matching algorithm is wrong when i does not backtrack after a failed match
in the middle. As a simple example data = OOORC, pattern= ORC, index = 1, this
algorithm will return -1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-1030) Java Tools Recover File command does not accurately find OrcFile.MAGIC
Yiqun Zhang created ORC-1030:
Summary: Java Tools Recover File command does not accurately find
OrcFile.MAGIC
Key: ORC-1030
URL: https://issues.apache.org/jira/browse/ORC-1030
Project: ORC
Issue Type: Bug
Components: Java, tools
Affects Versions: 1.6.11, 1.7.0, 1.8.0
Reporter: Yiqun Zhang
{code:java}
while (remaining > 0) {
int toRead = (int) Math.min(DEFAULT_BLOCK_SIZE, remaining);
byte[] data = new byte[toRead];
long startPos = corruptFileLen - remaining;
fdis.readFully(startPos, data, 0, toRead);
// find all MAGIC string and see if the file is readable from there
int index = 0;
long nextFooterOffset;
byte[] magicBytes = OrcFile.MAGIC.getBytes(StandardCharsets.UTF_8);
while (index != -1) {
index = indexOf(data, magicBytes, index + 1);
if (index != -1) {
nextFooterOffset = startPos + index + magicBytes.length + 1;
if (isReadable(corruptPath, conf, nextFooterOffset)) {
footerOffsets.add(nextFooterOffset);
}
}
}
System.err.println("Scanning for valid footers - startPos: " +
startPos +
" toRead: " + toRead + " remaining: " + remaining);
remaining = remaining - toRead;
}
{code}
Two adjacent reads may be exactly separated by OrcFile.MAGIC, making it
impossible to find the location of the recovered file. Because the current
implementation only matches in a single read.
{code:java}
private static int indexOf(final byte[] data, final byte[] pattern, final int
index) {
if (data == null || data.length == 0 || pattern == null || pattern.length
== 0 ||
index > data.length || index < 0) {
return -1;
}
int j = 0;
for (int i = index; i < data.length; i++) {
if (pattern[j] == data[i]) {
j++;
} else {
j = 0;
}
if (j == pattern.length) {
return i - pattern.length + 1;
}
}
return -1;
}
{code}
This matching algorithm is wrong when i does not backtrack after a failed match
in the middle. As a simple example data = OOORC, pattern= ORC, index = 1, this
algorithm will return -1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-1019) Remove redundant jackson dependencies
Yiqun Zhang created ORC-1019: Summary: Remove redundant jackson dependencies Key: ORC-1019 URL: https://issues.apache.org/jira/browse/ORC-1019 Project: ORC Issue Type: Improvement Components: build, Java Affects Versions: 1.8.0 Reporter: Yiqun Zhang Unfortunately, ORC-946 forgot to remove the bench dependency on jackson. In fact, the bench module does not directly depend on jackson, only spark indirectly depends on the specified version of jackson. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-1010) Bump tzdata from tzdata-2020e-1.tar.xz to tzdata-2021b-1.tar.xz
Yiqun Zhang created ORC-1010:
Summary: Bump tzdata from tzdata-2020e-1.tar.xz to
tzdata-2021b-1.tar.xz
Key: ORC-1010
URL: https://issues.apache.org/jira/browse/ORC-1010
Project: ORC
Issue Type: Test
Components: build
Affects Versions: 1.8.0
Reporter: Yiqun Zhang
Attachments: image-2021-09-28-11-25-18-160.png
{code:java}
CUSTOMBUILD : error : downloading
'ftp://cygwin.osuosl.org/pub/cygwin/noarch/release/tzdata/tzdata-2020e-1.tar.xz'
failed [C:\projects\orc\build\tzdata_ep.vcxproj]
status_code: 78
status_string: "Remote file not found"
log:
--- LOG BEGIN ---
timeout on name lookup is not supported
{code}
[https://cygwin.osuosl.org/noarch/release/tzdata/]
It looks like the archive tzdata-2020e-1.tar.xz has been deleted and
tzdata-2021b-1-src.tar.xz was created two days ago.
!image-2021-09-28-11-25-18-160.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-1008) Overflow detection code is incorrect in IntegerColumnStatisticsImpl
Yiqun Zhang created ORC-1008:
Summary: Overflow detection code is incorrect in
IntegerColumnStatisticsImpl
Key: ORC-1008
URL: https://issues.apache.org/jira/browse/ORC-1008
Project: ORC
Issue Type: Bug
Components: C++
Affects Versions: 1.7.0, 1.6.0, 1.8.0
Reporter: Yiqun Zhang
{code:java}
void IntegerColumnStatisticsImpl::update(int64_t value, int repetitions) {
_stats.updateMinMax(value);
if (_stats.hasSum()) {
bool wasPositive = _stats.getSum() >= 0;
_stats.setSum(value * repetitions + _stats.getSum());
if ((value >= 0) == wasPositive) {
_stats.setHasSum((_stats.getSum() >= 0) == wasPositive);
}
}
}
{code}
A counter-example can easily be given
Assume sum=1, update(std::numeric_limits::max(), 3);
value * repetitions + _stats.getSum() is overflowed, but is still a positive
number : 9223372036854775806
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-1007) Fix a warning from the shade plugin
Yiqun Zhang created ORC-1007:
Summary: Fix a warning from the shade plugin
Key: ORC-1007
URL: https://issues.apache.org/jira/browse/ORC-1007
Project: ORC
Issue Type: Bug
Components: build, Java
Affects Versions: 1.8.0
Reporter: Yiqun Zhang
build logs
{code:java}
Warning: Discovered module-info.class. Shading will break its strong
encapsulation.
Warning: Discovered module-info.class. Shading will break its strong
encapsulation.
Warning: Discovered module-info.class. Shading will break its strong
encapsulation.
{code}
I don't think the module system makes sense for uber.jar, it's useful during
development. The module-info.class of multiple modules will override each other
during the build of uber.jar, and I think its strong encapsulation may have an
effect on uber.jar.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-1006) Build and test in github workflow using the maven version specified in pom
Yiqun Zhang created ORC-1006: Summary: Build and test in github workflow using the maven version specified in pom Key: ORC-1006 URL: https://issues.apache.org/jira/browse/ORC-1006 Project: ORC Issue Type: Improvement Components: build, Java Affects Versions: 1.8.0 Reporter: Yiqun Zhang Build and test in github workflow using the maven version specified in pom. 1. Make the github workflow consistent with the build environment expressed in the readme. 2. Provides a capability: github workflow facilitates switching between maven versions and even supports build testing of multiple maven versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-1005) Ensure that the java and C++ implementations of determineEncoding in RunLengthIntegerWriterV2 are consistent.
Yiqun Zhang created ORC-1005: Summary: Ensure that the java and C++ implementations of determineEncoding in RunLengthIntegerWriterV2 are consistent. Key: ORC-1005 URL: https://issues.apache.org/jira/browse/ORC-1005 Project: ORC Issue Type: Improvement Components: C++, Java Affects Versions: 1.8.0 Reporter: Yiqun Zhang The current Java determineEncoding implementation is not optimal and may calculate ZigzagLiterals and zzBits100p before deciding on DELTA encoding, which is redundant. It is better to make the java implementation consistent with c++. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-995) Failed to build Java library
Yiqun Zhang created ORC-995:
---
Summary: Failed to build Java library
Key: ORC-995
URL: https://issues.apache.org/jira/browse/ORC-995
Project: ORC
Issue Type: Bug
Components: build, Java
Affects Versions: 1.8.0
Reporter: Yiqun Zhang
Failed to build Java library
{code:java}
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (default) @ orc-examples
---
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.orc:orc-shims:jar:1.8.0-SNAPSHOT:compile
[INFO]
[INFO] Reactor Summary for Apache ORC 1.8.0-SNAPSHOT:
[INFO]
[INFO] Apache ORC . SUCCESS [ 1.592 s]
[INFO] ORC Shims .. SUCCESS [ 1.590 s]
[INFO] ORC Core ... SUCCESS [01:49 min]
[INFO] ORC MapReduce .. SUCCESS [ 35.893 s]
[INFO] ORC Tools .. SUCCESS [ 12.179 s]
[INFO] ORC Examples ... FAILURE [ 2.986 s]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 02:44 min
[INFO] Finished at: 2021-09-15T10:43:44+08:00
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.1.2:analyze-only (default)
on project orc-examples: Dependency problems found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :orc-examples
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-992) Reached max repeat length, we can directly decide to use DELTA encoding
Yiqun Zhang created ORC-992:
---
Summary: Reached max repeat length, we can directly decide to use
DELTA encoding
Key: ORC-992
URL: https://issues.apache.org/jira/browse/ORC-992
Project: ORC
Issue Type: Improvement
Components: Java
Affects Versions: 1.7.0
Reporter: Yiqun Zhang
Fix For: 1.7.0
Reached max repeat length, we can directly decide to use DELTA encoding.
RunLengthIntegerWriterV2.java 756-760
{code:java}
// if fixed runs reached max repeat length then write values
if (fixedRunLength == MAX_SCOPE) {
determineEncoding();
writeValues();
}
{code}
If fixed runs reached max repeat length. We have been able to determine the use
of the DELTA code, fixedDelta is zero.
The computeZigZagLiterals, zzBits100p, and determine isFixedDelta within the
determineEncoding method are all redundant in the current case.
Similar practices.
RunLengthIntegerWriterV2.java 767-775
{code:java}
if (fixedRunLength >= MIN_REPEAT) {
if (fixedRunLength <= MAX_SHORT_REPEAT_LENGTH) {
encoding = EncodingType.SHORT_REPEAT;
} else {
encoding = EncodingType.DELTA;
isFixedDelta = true;
}
writeValues();
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-982) Extract checkstyle to a single file, help newcomers check code style
Yiqun Zhang created ORC-982: --- Summary: Extract checkstyle to a single file, help newcomers check code style Key: ORC-982 URL: https://issues.apache.org/jira/browse/ORC-982 Project: ORC Issue Type: Improvement Components: Java Affects Versions: 1.8.0 Reporter: Yiqun Zhang Fix For: 1.8.0 Attachments: screenshot-1.png Extract checkstyle to a single file, help newcomers check code style. [CheckStyle-IDEA|https://plugins.jetbrains.com/plugin/1065-checkstyle-idea] plugin is very simple to load this checkstyle.xml. This way you get checkstyle errors/warnings already when you are coding. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-976) Optimize compute zigZagLiterals
Yiqun Zhang created ORC-976:
---
Summary: Optimize compute zigZagLiterals
Key: ORC-976
URL: https://issues.apache.org/jira/browse/ORC-976
Project: ORC
Issue Type: Improvement
Components: Java
Affects Versions: 1.7.0
Reporter: Yiqun Zhang
Fix For: 1.7.0
{code:java}
private void computeZigZagLiterals() {
// populate zigzag encoded literals
long zzEncVal = 0;
for (int i = 0; i < numLiterals; i++) {
if (signed) {
zzEncVal = utils.zigzagEncode(literals[i]);
} else {
zzEncVal = literals[i];
}
zigzagLiterals[i] = zzEncVal;
}
}
{code}
Avoid conditional judgments in loops.
The unsigned case can use literals instead of zigzagLiterals. The number of
copies is proportional to the amount of user writes, which optimizes write
performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-975) Avoid double counting closestFixedBits in percentileBits method
Yiqun Zhang created ORC-975:
---
Summary: Avoid double counting closestFixedBits in percentileBits
method
Key: ORC-975
URL: https://issues.apache.org/jira/browse/ORC-975
Project: ORC
Issue Type: Improvement
Components: Java
Affects Versions: 1.7.0
Reporter: Yiqun Zhang
Fix For: 1.7.0
SerializationUtils
{code:java}
310 int idx = encodeBitWidth(findClosestNumBits(data[i]));
{code}
{code:java}
public int encodeBitWidth(int n) {
n = getClosestFixedBits(n);
.
}
{code}
{code:java}
public int findClosestNumBits(long value) {
final int numBits = 64 - Long.numberOfLeadingZeros(value);
return getClosestFixedBits(numBits);
}
{code}
getClosestFixedBits is called twice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-970) Reordering statements, improve readability
Yiqun Zhang created ORC-970: --- Summary: Reordering statements, improve readability Key: ORC-970 URL: https://issues.apache.org/jira/browse/ORC-970 Project: ORC Issue Type: Improvement Components: Java Affects Versions: 1.8.0 Reporter: Yiqun Zhang WriterImpl constructor logic is very confusing and needs to be refactored. For example, variable initialization and use do not go together. Checking logic does not precede heavy logic. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-958) Convert command support overwrite option
Yiqun Zhang created ORC-958: --- Summary: Convert command support overwrite option Key: ORC-958 URL: https://issues.apache.org/jira/browse/ORC-958 Project: ORC Issue Type: New Feature Components: Java, tools Affects Versions: 1.8.0 Reporter: Yiqun Zhang Fix For: 1.8.0 Add a small feature. Make the convert command support the overwrite option -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-946) Unified json library
Yiqun Zhang created ORC-946: --- Summary: Unified json library Key: ORC-946 URL: https://issues.apache.org/jira/browse/ORC-946 Project: ORC Issue Type: Improvement Components: Java Affects Versions: 1.8.0 Reporter: Yiqun Zhang Fix For: 1.8.0 The java project depends on several json libraries, can we unify them into one so that we can also reduce the dependencies. use jackson-core in orc-benchmarks-core org.apache.orc.bench.core.convert.json.JsonReader org.apache.orc.bench.core.convert.json.JsonWriter use gson in tools org.apache.orc.tools.convert.JsonReader org.apache.orc.tools.json.JsonSchemaFinder org.apache.orc.tools.json.JsonShredder org.apache.orc.tools.json.TestJsonSchemaFinder use jettjson in tools org.apache.orc.tools.KeyTool org.apache.orc.tools.JsonFileDump org.apache.orc.tools.PrintData I could provide a pr to unify the json library. But I'd like to hear from people 1. is the unification necessary? 2. if so, which library do we use? jackson/gson/jettison -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-937) Replace deprecated method
Yiqun Zhang created ORC-937: --- Summary: Replace deprecated method Key: ORC-937 URL: https://issues.apache.org/jira/browse/ORC-937 Project: ORC Issue Type: Improvement Reporter: Yiqun Zhang There are some deprecated method calls in the project that I think could be replaced by the recommended methods. org.apache.commons.cli OptionBuilder -> Option.builder GnuParser -> DefaultParser org.apache.parquet.avro AvroParquetWriter.builder(path) -> AvroParquetWriter.builder(outputFile) AvroParquetReader.builder(path) -> AvroParquetReader.builder(inputFile) DateColumnStatistics getMaximum -> getMaximumLocalDate getMinimum -> getMinimumLocalDate hadoop FSDataOutputStream(OutputStream) -> FSDataOutputStream(OutputStream, FileSystem.Statistics) fileStatus.isDir -> fileStatus.isDirectory -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-931) Optimize RunLengthIntegerWriterV2 code for better readability
Yiqun Zhang created ORC-931:
---
Summary: Optimize RunLengthIntegerWriterV2 code for better
readability
Key: ORC-931
URL: https://issues.apache.org/jira/browse/ORC-931
Project: ORC
Issue Type: Improvement
Reporter: Yiqun Zhang
RunLengthIntegerWriterV2.java
512-546 line
{code:java}
if (diffBitsLH > 1) {
for (int i = 0; i < numLiterals; i++) {
baseRedLiterals[i] = literals[i] - min;
}
brBits95p = utils.percentileBits(baseRedLiterals, 0, numLiterals, 0.95);
brBits100p = utils.percentileBits(baseRedLiterals, 0, numLiterals, 1.0);
if ((brBits100p - brBits95p) != 0 && Math.abs(min) < BASE_VALUE_LIMIT) {
encoding = EncodingType.PATCHED_BASE;
preparePatchedBlob();
return;
} else {
encoding = EncodingType.DIRECT;
return;
}
} else {
// if difference in bits between 95th percentile and 100th percentile is
// 0, then patch length will become 0. Hence we will fallback to direct
encoding = EncodingType.DIRECT;
return;
}
{code}
All three conditional branch logics have been completed and the return
statement is redundant.
691-704 line
{code:java}
if (fixedRunLength < MIN_REPEAT) {
variableRunLength = fixedRunLength;
fixedRunLength = 0;
determineEncoding();
writeValues();
} else if (fixedRunLength >= MIN_REPEAT
&& fixedRunLength <= MAX_SHORT_REPEAT_LENGTH) {
encoding = EncodingType.SHORT_REPEAT;
writeValues();
} else {
encoding = EncodingType.DELTA;
isFixedDelta = true;
writeValues();
}
{code}
fixedRunLength >= MIN_REPEAT is redundant, the previous condition already
ensures this. Extract the writeValues() method to the end. It seems better for
conditional judgements to deal only with encoding and state.
772-781 line
{code:java}
if (fixedRunLength >= MIN_REPEAT) {
if (fixedRunLength <= MAX_SHORT_REPEAT_LENGTH) {
encoding = EncodingType.SHORT_REPEAT;
writeValues();
} else {
encoding = EncodingType.DELTA;
isFixedDelta = true;
writeValues();
}
}
{code}
Ditto
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-927) Extracting duplicate codes for RowFilterBenchmark
Yiqun Zhang created ORC-927: --- Summary: Extracting duplicate codes for RowFilterBenchmark Key: ORC-927 URL: https://issues.apache.org/jira/browse/ORC-927 Project: ORC Issue Type: Improvement Components: Java Reporter: Yiqun Zhang There is a lot of duplicate code under the org.apache.orc.bench.hive.rowfilter package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ORC-904) Optimize userMetadata data structure to avoid traversal
Yiqun Zhang created ORC-904:
---
Summary: Optimize userMetadata data structure to avoid traversal
Key: ORC-904
URL: https://issues.apache.org/jira/browse/ORC-904
Project: ORC
Issue Type: Improvement
Components: Java
Reporter: Yiqun Zhang
ReaderImpl.java
{code:java}
@Override
public ByteBuffer getMetadataValue(String key) {
for(OrcProto.UserMetadataItem item: userMetadata) {
if (item.hasName() && item.getName().equals(key)) {
return item.getValue().asReadOnlyByteBuffer();
}
}
throw new IllegalArgumentException("Can't find user metadata " + key);
}
@Override
public boolean hasMetadataValue(String key) {
for(OrcProto.UserMetadataItem item: userMetadata) {
if (item.hasName() && item.getName().equals(key)) {
return true;
}
}
return false;
}
{code}
I think the data structure of userMetadata can be modified to map, avoid
traversal
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-902) The example of orc-example cannot be run
Yiqun Zhang created ORC-902:
---
Summary: The example of orc-example cannot be run
Key: ORC-902
URL: https://issues.apache.org/jira/browse/ORC-902
Project: ORC
Issue Type: Bug
Reporter: Yiqun Zhang
run orc-example, Throw an exception
{code:log}
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at
sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
{code}
mvn dependency:tree in orc-example
{code:java}
...
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:provided
...
{code}
Need to set scope to compile in orc-example/pom.xml
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-897) optimization loop termination condition in readerIsCompatible method
Yiqun Zhang created ORC-897:
---
Summary: optimization loop termination condition in
readerIsCompatible method
Key: ORC-897
URL: https://issues.apache.org/jira/browse/ORC-897
Project: ORC
Issue Type: Improvement
Components: Java
Reporter: Yiqun Zhang
{code:java}
private static boolean sameKeys(EncryptionKey[] first,
EncryptionKey[] next) {
if (first.length != next.length) {
return false;
}
for(int k = 0; k < first.length && k < next.length; ++k) {
if (!first[k].getKeyName().equals(next[k].getKeyName()) ||
first[k].getKeyVersion() != next[k].getKeyVersion() ||
first[k].getAlgorithm() != next[k].getAlgorithm()) {
return false;
}
}
return true;
}
{code}
{color:red}k < next.length{color} is not necessary
There are five similar codes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ORC-856) Fix exception description in findSubtype
Yiqun Zhang created ORC-856:
---
Summary: Fix exception description in findSubtype
Key: ORC-856
URL: https://issues.apache.org/jira/browse/ORC-856
Project: ORC
Issue Type: Bug
Components: Java
Affects Versions: 1.7.0, master
Reporter: Yiqun Zhang
Fix For: 1.7.0
{code:java}
public static void findSubtype(TypeDescription schema,
int goal,
TypeVisitor visitor) {
TypeDescription current = schema;
int id = schema.getId();
if (goal < id || goal > schema.getMaximumId()) {
throw new IllegalArgumentException("Unknown type id " + id + " in " +
current.toJson());
}
}
{code}
"Unknown type id " + {color:red}id{color} + " in " + current.toJson()
Need to be modified to goal here
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
