Re: Travis CI jobs gummed up on Arrow PRs?
I have just reported this issue at the TravisCI forum. https://travis-ci.community/t/s390x-jobs-have-not-been-almost-executed/10581 Regards, Kazuaki Ishizaki, Sutou Kouhei wrote on 2020/11/16 10:02:18: > From: Sutou Kouhei > To: dev@arrow.apache.org > Date: 2020/11/16 10:02 > Subject: [EXTERNAL] Re: Travis CI jobs gummed up on Arrow PRs? > > Hi, > > > 1. Is anyone else knows about these failures? > > "these failures" means that the Travis CI jobs aren't ran, > right? (It doesn't mean that the Travis CI jobs reports > "failure".) > > This may be a Travis CI bug. > > > 2. Should we look into disabling these checks for PRs that only touch rust > > code? I can do this, but am not sure of the history > > One approach is adding "[travis skip]" to commit message. > We can't use path based conditional build on Travis CI > because Travis CI doesn't provide the feature. > > See also: > > * INVALID URI REMOVED > u=https-3A__docs.travis-2Dci.com_user_conditions-2Dv1=DwICAg=jf_iaSHvJObTbx- > siA1ZOg=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg=U4B_SrhIun- > kKemWyzO5XUEZhrCIfFKnR967kBAI3rE=iGmV- > uUdAjCFprWPStYvU1jHvMt7D2YM8hmtdRLC4n4= > * INVALID URI REMOVED > u=https-3A__docs.travis-2Dci.com_user_conditions-2Dtesting=DwICAg=jf_iaSHvJObTbx- > siA1ZOg=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg=U4B_SrhIun- > kKemWyzO5XUEZhrCIfFKnR967kBAI3rE=yO4rf79YYHbt8NJKOLGaCKMKUehCe_ub2RtpDLq06QA= > > > Thanks, > -- > kou > > In > "Travis CI jobs gummed up on Arrow PRs?" on Sun, 15 Nov 2020 07:55:27 -0500, > Andrew Lamb wrote: > > > Sorry if this has already been discussed. > > > > There seems to be something wrong with the Travis CI jobs on some Arrow PRs > > -- for example, > > INVALID URI REMOVED > u=https-3A__github.com_apache_arrow_pull_8662_checks-3Fcheck-5Frun-5Fid-3D1400052607=DwICAg=jf_iaSHvJObTbx- > siA1ZOg=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg=U4B_SrhIun- > kKemWyzO5XUEZhrCIfFKnR967kBAI3rE=V6iophseGVoYXwP1pLHBNtuMiHnEjT2bj69- > iULRCZc= . > > They go into the "pending" state and never seem to actually run. > > > > Since these appear to be checks of the C++ implementation on s390 or ARM, > > which aren't relevant to the Rust implementation, it isn't really blocking > > anything. > > > > I am wondering > > 1. Is anyone else knows about these failures? > > 2. Should we look into disabling these checks for PRs that only touch rust > > code? I can do this, but am not sure of the history > > > > Thank you, > > Andrew > > > > p.s. here is another example of a non rust PR that seems to show the same > > issue: > > INVALID URI REMOVED > u=https-3A__github.com_apache_arrow_pull_8647=DwICAg=jf_iaSHvJObTbx- > siA1ZOg=b70dG_9wpCdZSkBJahHYQ4IwKMdp2hQM29f-ZCGj9Pg=U4B_SrhIun- > kKemWyzO5XUEZhrCIfFKnR967kBAI3rE=lA3pYfzvwvn9bMYt2saPIQe5ZmC11vkGCPO8g6TyMMM= > > > > p.p.s. Here is a screenshot showing the travis CI UI for such a gummed up > > test > > > > [image: Screen Shot 2020-11-15 at 7.50.27 AM.png] >
Re: Travis CI jobs gummed up on Arrow PRs?
Hi, > 1. Is anyone else knows about these failures? "these failures" means that the Travis CI jobs aren't ran, right? (It doesn't mean that the Travis CI jobs reports "failure".) This may be a Travis CI bug. > 2. Should we look into disabling these checks for PRs that only touch rust > code? I can do this, but am not sure of the history One approach is adding "[travis skip]" to commit message. We can't use path based conditional build on Travis CI because Travis CI doesn't provide the feature. See also: * https://docs.travis-ci.com/user/conditions-v1 * https://docs.travis-ci.com/user/conditions-testing Thanks, -- kou In "Travis CI jobs gummed up on Arrow PRs?" on Sun, 15 Nov 2020 07:55:27 -0500, Andrew Lamb wrote: > Sorry if this has already been discussed. > > There seems to be something wrong with the Travis CI jobs on some Arrow PRs > -- for example, > https://github.com/apache/arrow/pull/8662/checks?check_run_id=1400052607. > They go into the "pending" state and never seem to actually run. > > Since these appear to be checks of the C++ implementation on s390 or ARM, > which aren't relevant to the Rust implementation, it isn't really blocking > anything. > > I am wondering > 1. Is anyone else knows about these failures? > 2. Should we look into disabling these checks for PRs that only touch rust > code? I can do this, but am not sure of the history > > Thank you, > Andrew > > p.s. here is another example of a non rust PR that seems to show the same > issue: > https://github.com/apache/arrow/pull/8647 > > p.p.s. Here is a screenshot showing the travis CI UI for such a gummed up > test > > [image: Screen Shot 2020-11-15 at 7.50.27 AM.png]
Re: [DISCUSS] Alternative design for KMS interaction in parquet-cpp
@Gidon Copied the gist to a google doc for commenting: https://docs.google.com/document/d/11qz84ajysvVo5ZAV9mXKOeh6ay4-xgkBrubggCP5220/edit# @Micah > it would be preferable to put them in an internal namespace to ensure adequate unit testing is in place. Agreed; my intent was only to indicate that implementation details should be private. We can even have *_internal.h header files for those to make them available for unit testing as we do with some other classes. I've added a note of this caveat to the doc. > It seems like KmsClient and maybe PropertiesDrivenCryptoFactory should still be implemented > with internal synchronization to guarantee thread-safety? If these are exclusively accessed from the main thread (where they are used to assemble FileDecryptionProperties) then there is still no need to synchronize them. @Tham > As I see, the returned FileDecryptionProperties object from > PropertiesDrivenCryptoFactory is not mutable, > since it has a DecryptionKeyRetriever object. If the DecryptionKeyRetriever accesses the caches directly, then it will indeed need to be synchronized. Instead, we can ensure that cache access happens on the main thread before worker tasks are launched. After we assemble this ahead-of-time set of keys it will not change during the course of a read, so the DecryptionKeyRetriever can safely access it without mutexes. I've added a comment to the doc On Fri, Nov 13, 2020 at 3:09 AM Gidon Gershinsky wrote: > Hi all, > > Glad to see the parquet-cpp progress on this! Can I suggest creating a > googledoc for the technical discussion? The current md doc format seems to > be harder for pinpointed comments. I got a few, but they are too minor for > sending to the two mailing lists. > > Cheers, Gidon > > > On Fri, Nov 13, 2020 at 9:17 AM Tham Ha wrote: > >> Hi Ben, >> >> Can you reconsider this point: >> > Properties (including FileDecryptionProperties) are immutable after >> construction and thus can be safely shared between threads with no further >> synchronization. >> >> As I see, the returned FileDecryptionProperties object from >> PropertiesDrivenCryptoFactory >> is not mutable, since it has a DecryptionKeyRetriever object (i.e. >> FileKeyUnwrapper >> object in this pull) that helps get the key from key metadata. >> From the current implementation of FileKeyUnwrapper with the cache, I >> think synchronization is necessary, unless you have another idea. >> >> Cheers, Tham >> >> On Fri, Nov 13, 2020 at 1:28 PM Micah Kornfield >> wrote: >> >>> I skimmed through and this seems like a clean design (I would have to >>> reread the PR to do a comparison. A few thoughts of the top of my head: >>> >>> >>> > - Multiple internal classes are left public in header files, where it >>> > would be >>> > preferred that public classes be kept to a minimum. >>> >>> I think some of these classes are non-trivial. If that is the case it >>> would be preferable to put them in an internal namespace to ensure >>> adequate unit testing is in place. >>> >>> It seems like KmsClient and maybe PropertiesDrivenCryptoFactory should >>> still be implemented with internal synchronization to guarantee >>> thread-safety? >>> >>> >>> >>> On Wed, Nov 11, 2020 at 6:19 PM Benjamin Kietzman >>> wrote: >>> >>> > In the context of https://issues.apache.org/jira/browse/ARROW-9318 / >>> > https://github.com/apache/arrow/pull/8023 which port the parquet-mr >>> > design to c++: there's some question whether that design is consistent >>> > with the style and conventions of the c++ implementation of parquet. >>> > >>> > Here is a Gist with a sketched alternative design adapted to c++ >>> > https://gist.github.com/bkietz/f701d32add6f5a2aeed89ce36a443d43 >>> > >>> > Specific concerns in brief: >>> > - Multiple internal classes are left public in header files, where it >>> > would be >>> > preferred that public classes be kept to a minimum. >>> > - Several levels of explicit synchronization with mutexes are used to >>> > guarantee >>> > that each class is completely thread safe, but this is unnecessary >>> > overhead >>> > given the pattern of use of `parquet-cpp`. >>> > >>> >> >> >> -- >> Tham Ha Thi >> C++ Developer >> *EMOTIV* www.emotiv.com >> Neurotech for the Global Community >> >> LEGAL NOTICE >> >> This message (including all attachments) contains confidential >> information intended for a specific individual and purpose, and is >> protected by law. Any confidentiality or privilege is not waived or lost >> because this email has been sent to you by mistake. If you have received it >> in error, please let us know by reply email, delete it from your system and >> destroy any copies. >> >> This email is also subject to copyright. Any disclosure, copying, or >> distribution of this message, or the taking of any action based on it, is >> strictly prohibited. >> >> Emails may be interfered with, may contain computer viruses or other >> defects and may not be successfully replicated on other systems. We
Travis CI jobs gummed up on Arrow PRs?
Sorry if this has already been discussed. There seems to be something wrong with the Travis CI jobs on some Arrow PRs -- for example, https://github.com/apache/arrow/pull/8662/checks?check_run_id=1400052607. They go into the "pending" state and never seem to actually run. Since these appear to be checks of the C++ implementation on s390 or ARM, which aren't relevant to the Rust implementation, it isn't really blocking anything. I am wondering 1. Is anyone else knows about these failures? 2. Should we look into disabling these checks for PRs that only touch rust code? I can do this, but am not sure of the history Thank you, Andrew p.s. here is another example of a non rust PR that seems to show the same issue: https://github.com/apache/arrow/pull/8647 p.p.s. Here is a screenshot showing the travis CI UI for such a gummed up test [image: Screen Shot 2020-11-15 at 7.50.27 AM.png]
[NIGHTLY] Arrow Build Report for Job nightly-2020-11-15-0
Arrow Build Report for Job nightly-2020-11-15-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0 Failed Tasks: - conda-win-vs2017-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-win-vs2017-py36 - conda-win-vs2017-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-win-vs2017-py37 - test-conda-python-3.7-spark-branch-3.0: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-test-conda-python-3.7-spark-branch-3.0 - test-conda-python-3.8-jpype: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-test-conda-python-3.8-jpype - test-ubuntu-18.04-docs: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-test-ubuntu-18.04-docs Succeeded Tasks: - centos-6-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-centos-6-amd64 - centos-7-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-centos-7-aarch64 - centos-7-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-centos-7-amd64 - centos-8-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-centos-8-aarch64 - centos-8-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-centos-8-amd64 - conda-clean: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-clean - conda-linux-gcc-py36-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-drone-conda-linux-gcc-py36-aarch64 - conda-linux-gcc-py36-cpu: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-linux-gcc-py36-cpu - conda-linux-gcc-py36-cuda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-linux-gcc-py36-cuda - conda-linux-gcc-py37-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-drone-conda-linux-gcc-py37-aarch64 - conda-linux-gcc-py37-cpu: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-linux-gcc-py37-cpu - conda-linux-gcc-py37-cuda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-linux-gcc-py37-cuda - conda-linux-gcc-py38-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-drone-conda-linux-gcc-py38-aarch64 - conda-linux-gcc-py38-cpu: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-linux-gcc-py38-cpu - conda-linux-gcc-py38-cuda: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-linux-gcc-py38-cuda - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-osx-clang-py38 - conda-win-vs2017-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-azure-conda-win-vs2017-py38 - debian-buster-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-debian-buster-amd64 - debian-buster-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-debian-buster-arm64 - debian-stretch-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-debian-stretch-amd64 - debian-stretch-arm64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-debian-stretch-arm64 - example-cpp-minimal-build-static-system-dependency: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-example-cpp-minimal-build-static-system-dependency - example-cpp-minimal-build-static: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-github-example-cpp-minimal-build-static - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-gandiva-jar-osx - gandiva-jar-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-gandiva-jar-xenial - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-homebrew-cpp - homebrew-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-11-15-0-travis-homebrew-r-autobrew - nuget:
