Re: Running all tests via Crave.io

[Mikhail Khludnev]

Thank you, all.
Build passed!

On Thu, Nov 23, 2023 at 12:50 AM David Smiley 
wrote:

> Mikhail,
>
> This is tracked here: https://issues.apache.org/jira/browse/SOLR-17081
> (thanks Christine) and I've updated it with further details.
> I've been in contact with UV.  The build image has been updated with fonts
> and recent builds have succeeded proof:
> https://github.com/apache/solr/actions/workflows/tests-via-crave.yml
> So I think I can close the issue.
>
> ~ David
>
>
> On Wed, Nov 22, 2023 at 3:43 PM Mikhail Khludnev  wrote:
>
> > Hello Yuvraaj,
> > Recently Crave became prone to specific bug.
> >
> >
> https://github.com/apache/solr/actions/runs/6909103380/job/18849339883?pr=2046
> >
> > java.lang.RuntimeException: Fontconfig head is null, check your fonts or
> > fonts configuration
> >  at
> >
> >
> java.desktop/sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1271)
> >
> > Excel test (yes it is. TestXLSXResponseWriter) attempts to measure font
> > size and fails on a headless jvm (I assume). I don't think we changed
> > anything related to it recently.
> > The problem is that it breaks one module and a whole build, so we have no
> > tests for PRs.
> > The obvious fix is to pass java.awt.headless=true property
> > And it's exactly what we do in  gradle/testing/defaults-tests.gradle
> > When I turn debug locally I see that Test Executor receives this property
> > via -D command argument.
> > Even more than that, I setup headless jdk locally, and even commented
> this
> > row in defaults-tests.gradle and test passed anyway.
> > I'm out of ideas.
> > Can you recommend how to resolve this issue?
> > Appreciate your advice.
> >
> >
> >
> > On Sat, Mar 11, 2023 at 6:06 AM Shawn Heisey 
> wrote:
> >
> > > On 3/10/23 06:49, Kevin Risden wrote:
> > > > So in the case of https://github.com/apache/solr/pull/571 that would
> > be
> > > > concretely:
> > > >
> > > > git fetch origin pull/571/head:pr-571
> > > > git checkout pr-571
> > > >
> > > > This would work if https://github.com/apache/solr is cloned. There
> > > should
> > > > be no need to try to find a merge base and reapply commits.
> > >
> > > Even a full re-clone shouldn't take all that long.  I would hope Crave
> > > has a lot more bandwidth than I do.  38 seconds for me to clone the
> repo
> > > from github:
> > >
> > > ---
> > > sheisey@sheisey-desktop:~/fff$ time git clone
> > > https://github.com/apache/solr.git
> > > Cloning into 'solr'...
> > > remote: Enumerating objects: 1254736, done.
> > > remote: Counting objects: 100% (667/667), done.
> > > remote: Compressing objects: 100% (351/351), done.
> > > remote: Total 1254736 (delta 256), reused 535 (delta 162), pack-reused
> > > 1254069
> > > Receiving objects: 100% (1254736/1254736), 462.65 MiB | 20.66 MiB/s,
> > done.
> > > Resolving deltas: 100% (739402/739402), done.
> > >
> > > real0m38.137s
> > > user1m24.712s
> > > sys 0m8.445s
> > > ---
> > >
> > > If the repo is already cloned, it should be very quick to reset the
> > > repository and repo to a completely clean state on the main branch, do
> a
> > > git pull, and then do the steps you outlined above to get the PR.  Not
> > > that I know what the right git commands are to do these things.
> > >
> > > Thanks,
> > > Shawn
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > For additional commands, e-mail: dev-h...@solr.apache.org
> > >
> > >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>


-- 
Sincerely yours
Mikhail Khludnev

Re: Running all tests via Crave.io

[David Smiley]

Mikhail,

This is tracked here: https://issues.apache.org/jira/browse/SOLR-17081
(thanks Christine) and I've updated it with further details.
I've been in contact with UV.  The build image has been updated with fonts
and recent builds have succeeded proof:
https://github.com/apache/solr/actions/workflows/tests-via-crave.yml
So I think I can close the issue.

~ David


On Wed, Nov 22, 2023 at 3:43 PM Mikhail Khludnev  wrote:

> Hello Yuvraaj,
> Recently Crave became prone to specific bug.
>
> https://github.com/apache/solr/actions/runs/6909103380/job/18849339883?pr=2046
>
> java.lang.RuntimeException: Fontconfig head is null, check your fonts or
> fonts configuration
>  at
>
> java.desktop/sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1271)
>
> Excel test (yes it is. TestXLSXResponseWriter) attempts to measure font
> size and fails on a headless jvm (I assume). I don't think we changed
> anything related to it recently.
> The problem is that it breaks one module and a whole build, so we have no
> tests for PRs.
> The obvious fix is to pass java.awt.headless=true property
> And it's exactly what we do in  gradle/testing/defaults-tests.gradle
> When I turn debug locally I see that Test Executor receives this property
> via -D command argument.
> Even more than that, I setup headless jdk locally, and even commented this
> row in defaults-tests.gradle and test passed anyway.
> I'm out of ideas.
> Can you recommend how to resolve this issue?
> Appreciate your advice.
>
>
>
> On Sat, Mar 11, 2023 at 6:06 AM Shawn Heisey  wrote:
>
> > On 3/10/23 06:49, Kevin Risden wrote:
> > > So in the case of https://github.com/apache/solr/pull/571 that would
> be
> > > concretely:
> > >
> > > git fetch origin pull/571/head:pr-571
> > > git checkout pr-571
> > >
> > > This would work if https://github.com/apache/solr is cloned. There
> > should
> > > be no need to try to find a merge base and reapply commits.
> >
> > Even a full re-clone shouldn't take all that long.  I would hope Crave
> > has a lot more bandwidth than I do.  38 seconds for me to clone the repo
> > from github:
> >
> > ---
> > sheisey@sheisey-desktop:~/fff$ time git clone
> > https://github.com/apache/solr.git
> > Cloning into 'solr'...
> > remote: Enumerating objects: 1254736, done.
> > remote: Counting objects: 100% (667/667), done.
> > remote: Compressing objects: 100% (351/351), done.
> > remote: Total 1254736 (delta 256), reused 535 (delta 162), pack-reused
> > 1254069
> > Receiving objects: 100% (1254736/1254736), 462.65 MiB | 20.66 MiB/s,
> done.
> > Resolving deltas: 100% (739402/739402), done.
> >
> > real0m38.137s
> > user1m24.712s
> > sys 0m8.445s
> > ---
> >
> > If the repo is already cloned, it should be very quick to reset the
> > repository and repo to a completely clean state on the main branch, do a
> > git pull, and then do the steps you outlined above to get the PR.  Not
> > that I know what the right git commands are to do these things.
> >
> > Thanks,
> > Shawn
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > For additional commands, e-mail: dev-h...@solr.apache.org
> >
> >
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: Running all tests via Crave.io

[Mikhail Khludnev]

Hello Yuvraaj,
Recently Crave became prone to specific bug.
https://github.com/apache/solr/actions/runs/6909103380/job/18849339883?pr=2046

java.lang.RuntimeException: Fontconfig head is null, check your fonts or
fonts configuration
 at
java.desktop/sun.awt.FontConfiguration.getVersion(FontConfiguration.java:1271)

Excel test (yes it is. TestXLSXResponseWriter) attempts to measure font
size and fails on a headless jvm (I assume). I don't think we changed
anything related to it recently.
The problem is that it breaks one module and a whole build, so we have no
tests for PRs.
The obvious fix is to pass java.awt.headless=true property
And it's exactly what we do in  gradle/testing/defaults-tests.gradle
When I turn debug locally I see that Test Executor receives this property
via -D command argument.
Even more than that, I setup headless jdk locally, and even commented this
row in defaults-tests.gradle and test passed anyway.
I'm out of ideas.
Can you recommend how to resolve this issue?
Appreciate your advice.



On Sat, Mar 11, 2023 at 6:06 AM Shawn Heisey  wrote:

> On 3/10/23 06:49, Kevin Risden wrote:
> > So in the case of https://github.com/apache/solr/pull/571 that would be
> > concretely:
> >
> > git fetch origin pull/571/head:pr-571
> > git checkout pr-571
> >
> > This would work if https://github.com/apache/solr is cloned. There
> should
> > be no need to try to find a merge base and reapply commits.
>
> Even a full re-clone shouldn't take all that long.  I would hope Crave
> has a lot more bandwidth than I do.  38 seconds for me to clone the repo
> from github:
>
> ---
> sheisey@sheisey-desktop:~/fff$ time git clone
> https://github.com/apache/solr.git
> Cloning into 'solr'...
> remote: Enumerating objects: 1254736, done.
> remote: Counting objects: 100% (667/667), done.
> remote: Compressing objects: 100% (351/351), done.
> remote: Total 1254736 (delta 256), reused 535 (delta 162), pack-reused
> 1254069
> Receiving objects: 100% (1254736/1254736), 462.65 MiB | 20.66 MiB/s, done.
> Resolving deltas: 100% (739402/739402), done.
>
> real0m38.137s
> user1m24.712s
> sys 0m8.445s
> ---
>
> If the repo is already cloned, it should be very quick to reset the
> repository and repo to a completely clean state on the main branch, do a
> git pull, and then do the steps you outlined above to get the PR.  Not
> that I know what the right git commands are to do these things.
>
> Thanks,
> Shawn
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> For additional commands, e-mail: dev-h...@solr.apache.org
>
>

-- 
Sincerely yours
Mikhail Khludnev

Re: Running all tests via Crave.io

[Shawn Heisey]

On 3/10/23 06:49, Kevin Risden wrote:

So in the case of https://github.com/apache/solr/pull/571 that would be
concretely:

git fetch origin pull/571/head:pr-571
git checkout pr-571

This would work if https://github.com/apache/solr is cloned. There should
be no need to try to find a merge base and reapply commits.


Even a full re-clone shouldn't take all that long.  I would hope Crave 
has a lot more bandwidth than I do.  38 seconds for me to clone the repo 
from github:


---
sheisey@sheisey-desktop:~/fff$ time git clone 
https://github.com/apache/solr.git

Cloning into 'solr'...
remote: Enumerating objects: 1254736, done.
remote: Counting objects: 100% (667/667), done.
remote: Compressing objects: 100% (351/351), done.
remote: Total 1254736 (delta 256), reused 535 (delta 162), pack-reused 
1254069

Receiving objects: 100% (1254736/1254736), 462.65 MiB | 20.66 MiB/s, done.
Resolving deltas: 100% (739402/739402), done.

real0m38.137s
user1m24.712s
sys 0m8.445s
---

If the repo is already cloned, it should be very quick to reset the 
repository and repo to a completely clean state on the main branch, do a 
git pull, and then do the steps you outlined above to get the PR.  Not 
that I know what the right git commands are to do these things.


Thanks,
Shawn

-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Running all tests via Crave.io

[Yuvraaj Kelkar]

Disclaimer: technical details that aren't strictly about Solr coming up ahead:


Crave builds and tests the Solr code on a remote server.
The first step to doing that is to "clone the exact sources on the remote 
system".
A brute-force way to do that would be rsync the sources from the Github runner 
to the remote server, but this would involve a very large amount of data being 
rsync'd - which is usually a lot of time: Easily 10 minutes for Solr.
Instead of the brute-force, Crave uses the following method:
Find out a git commit ID (in the Solr git repo) that is "guaranteed to be known 
to any clone": This is what we call the merge-base - because that's what a 
certain git bash completion script called it.

Collect a git patch of the PR w.r.t that merge base. Send this patch across the 
wire to the remote server.

git checkout  then git am the patch on the remote server. If this 
sounds like the old fashioned Gerrit or LKML-style or Lore-style patch 
workflow, you're right. it is exactly that

Start the build/test on the remote server

The reason for this method instead of plain rsync is that it reduces the amount 
of data to be transferred from being often GBs (and in some other projects, 
100s of GBs) to a few KB.
O(n) -> O(k) space.
The time required also drops considerably: usually completed in 10-20 seconds.

Of course, I'm sure there's improvements possible: We're open to suggestions.


Thanks,
-Uv

On Mar 10 2023, at 5:49 am, Kevin Risden  wrote:
> I guess I'm not quite sure why Crave is trying to find the merge base...
> The PR is actually a branch in git - it has the full state of the repo that
> should be tested. This is accomplished basically by the following [1]
>
> git fetch origin pull/ID/head:BRANCH_NAME
> git checkout BRANCH_NAME
>
> So in the case of https://github.com/apache/solr/pull/571 that would be
> concretely:
>
> git fetch origin pull/571/head:pr-571
> git checkout pr-571
>
> This would work if https://github.com/apache/solr is cloned. There should
> be no need to try to find a merge base and reapply commits.
>
> [1]
> https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
>
> Kevin Risden
>
> On Thu, Mar 9, 2023 at 8:14 PM Yuvraaj Kelkar  wrote:
> > Upvote for Mikhail's suggestion: That's the root cause of the problem: The
> > PR has drifted too much for Crave to get a correct git merge base.
> > Evidence : The PR is at most 8 files. Crave seems to thing it is 42 files
> > and thousands of changes:
> > Run ./crave run --clean
> > 42 files changed, 1032 insertions(+), 261 deletions(-)
> > Thanks,
> > -Uv
> > On Mar 9 2023, at 9:46 am, Mikhail Khludnev  wrote:
> > > Hello Kevin,
> > > It happened to me too. I fixed it with rebasing pr branch.
> > > $git pull --rebase origin main
> > > $git push -f myfork PR-branch
> > >
> > > On Thu, Mar 9, 2023 at 5:50 PM Kevin Risden  wrote:
> > > > I've noticed failures like this on PRs quite a few times when github
> > > > actions for `./gradlew check` pass no issue.
> > > >
> > > > https://github.com/apache/solr/actions/runs/4375627158/jobs/7656583100
> > > >
> > > > Setting up workspace (this could take some time)...
> > > > > Pulling container image...
> > > > > Finished pulling container image
> > > > > Switched to a new branch
> > > > > 'detached-57bb78866cc07e76f6d16cc0e08e1b6ef8701444'
> > > > > setting commitID to 90c4c50877c81fdf578c8ccf8f6976408eefaf27
> > > > > committer id not set in git config, hence setting it to Crave
> > > > > committer email not set in git config, hence setting it to
> > > > cr...@crave.io
> > > > > error: patch failed:
> > > > > solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:48
> > > > > error:
> > solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:
> > > > > patch does not apply
> > > > > hint: Use 'git am --show-current-patch=diff' to see the failed patch
> > > > > Applying: SOLR-16681: Throw exception when attempting to replace
> > > > uniqueKey
> > > > > via fl in distributed request (#1384)
> > > > > Applying: SOLR-16676: Test improvements
> > > > > Applying: Avoid assertNotNull
> > > > > Applying: SOLR-16643: reRankOperator=multiply/replace options (#1431)
> > > > > Applying: SOLR-16676 Http2SolrClient loss of MDC context flaky test
> > > > changes
> > > > > Patch failed at 0005 SOLR-16676 Http2SolrClient loss of MDC context
> > flaky
> > > > > test changes
> > > > > When you have resolved this problem, run "git am --continue".
> > > > > If you prefer to skip this patch, run "git am --skip" instead.
> > > > > To restore the original branch and stop patching, run "git am
> > --abort".
> > > >
> > > >
> > > > The branch has main merged and is up to date so there should be no
> > > > conflicts. Based on the output its hard to tell why it failed to apply.
> > > >
> > > > Kevin Risden
> > > >
> > > >
> > > > On Thu, Feb 9, 2023 at 4:33 PM Jan Høydahl 
> > wrote:
> > > >
> > > > > Looks lik

Re: Running all tests via Crave.io

[Kevin Risden]

I guess I'm not quite sure why Crave is trying to find the merge base...
The PR is actually a branch in git - it has the full state of the repo that
should be tested. This is accomplished basically by the following [1]

git fetch origin pull/ID/head:BRANCH_NAME
git checkout BRANCH_NAME

So in the case of https://github.com/apache/solr/pull/571 that would be
concretely:

git fetch origin pull/571/head:pr-571
git checkout pr-571

This would work if https://github.com/apache/solr is cloned. There should
be no need to try to find a merge base and reapply commits.

[1]
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Kevin Risden


On Thu, Mar 9, 2023 at 8:14 PM Yuvraaj Kelkar  wrote:

> Upvote for Mikhail's suggestion: That's the root cause of the problem: The
> PR has drifted too much for Crave to get a correct git merge base.
> Evidence : The PR is at most 8 files. Crave seems to thing it is 42 files
> and thousands of changes:
> Run ./crave run --clean
> 42 files changed, 1032 insertions(+), 261 deletions(-)
> Thanks,
> -Uv
> On Mar 9 2023, at 9:46 am, Mikhail Khludnev  wrote:
> > Hello Kevin,
> > It happened to me too. I fixed it with rebasing pr branch.
> > $git pull --rebase origin main
> > $git push -f myfork PR-branch
> >
> > On Thu, Mar 9, 2023 at 5:50 PM Kevin Risden  wrote:
> > > I've noticed failures like this on PRs quite a few times when github
> > > actions for `./gradlew check` pass no issue.
> > >
> > > https://github.com/apache/solr/actions/runs/4375627158/jobs/7656583100
> > >
> > > Setting up workspace (this could take some time)...
> > > > Pulling container image...
> > > > Finished pulling container image
> > > > Switched to a new branch
> > > > 'detached-57bb78866cc07e76f6d16cc0e08e1b6ef8701444'
> > > > setting commitID to 90c4c50877c81fdf578c8ccf8f6976408eefaf27
> > > > committer id not set in git config, hence setting it to Crave
> > > > committer email not set in git config, hence setting it to
> > > cr...@crave.io
> > > > error: patch failed:
> > > > solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:48
> > > > error:
> solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:
> > > > patch does not apply
> > > > hint: Use 'git am --show-current-patch=diff' to see the failed patch
> > > > Applying: SOLR-16681: Throw exception when attempting to replace
> > > uniqueKey
> > > > via fl in distributed request (#1384)
> > > > Applying: SOLR-16676: Test improvements
> > > > Applying: Avoid assertNotNull
> > > > Applying: SOLR-16643: reRankOperator=multiply/replace options (#1431)
> > > > Applying: SOLR-16676 Http2SolrClient loss of MDC context flaky test
> > > changes
> > > > Patch failed at 0005 SOLR-16676 Http2SolrClient loss of MDC context
> flaky
> > > > test changes
> > > > When you have resolved this problem, run "git am --continue".
> > > > If you prefer to skip this patch, run "git am --skip" instead.
> > > > To restore the original branch and stop patching, run "git am
> --abort".
> > >
> > >
> > > The branch has main merged and is up to date so there should be no
> > > conflicts. Based on the output its hard to tell why it failed to apply.
> > >
> > > Kevin Risden
> > >
> > >
> > > On Thu, Feb 9, 2023 at 4:33 PM Jan Høydahl 
> wrote:
> > >
> > > > Looks like crave test (in github actions) is still somewhat unstable,
> > > > sometimes failing to allocate resoources?
> > > >
> > > > Also, question regarding how to inspect crave failed tests. How to
> access
> > > > the full log, it just says it is at file:///tmp/... but is there a
> way
> > > for
> > > > crave to generate a URL where one can access the test result output
> in
> > > the
> > > > cloud?
> > > >
> > > > Jan
> > > >
> > > > > 29. jan. 2023 kl. 20:32 skrev Yuvraaj Kelkar :
> > > > >
> > > > > Updated default command to
> > > > > ./gradlew localSettings && ./gradlew --max-workers=`nproc`
> > > > -Ptests.jvms=96 test
> > > > >
> > > > > Thanks,
> > > > > -Uv
> > > > >
> > > > > On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya <
> > > > ichattopadhy...@gmail.com> wrote:
> > > > >> Thanks to Crave! Seems like there's Dual XEON 24 core processors
> > > > >> available on the system.
> > > > >>
> > > > >> I pushed the parallelism to 96, and times reduced from 8.5 min to
> ~5.5
> > > > min.
> > > > >> ~/crave run -- './gradlew localSettings && ./gradlew
> > > > >> --max-workers=`nproc` -Ptests.jvms=96 test'
> > > > >> 
> > > > >> The slowest tests (exceeding 500 ms) during this run:
> > > > >> 94.39s BasicDistributedZkTest.test (:solr:core)
> > > > >> 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
> > > > >> 74.73s PeerSyncReplicationTest.test (:solr:core)
> > > > >> 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
> > > > >> (:solr:core)
> > > > >> 59.15s ReplicationFactorTest.test (:solr:core)
> > > > >> 58.82s HdfsDirectoryTest.testRandomAccessWrites
> (:solr:modules:hdf

Re: Running all tests via Crave.io

[Yuvraaj Kelkar]

Upvote for Mikhail's suggestion: That's the root cause of the problem: The PR 
has drifted too much for Crave to get a correct git merge base.
Evidence : The PR is at most 8 files. Crave seems to thing it is 42 files and 
thousands of changes:
Run ./crave run --clean
42 files changed, 1032 insertions(+), 261 deletions(-)
Thanks,
-Uv
On Mar 9 2023, at 9:46 am, Mikhail Khludnev  wrote:
> Hello Kevin,
> It happened to me too. I fixed it with rebasing pr branch.
> $git pull --rebase origin main
> $git push -f myfork PR-branch
>
> On Thu, Mar 9, 2023 at 5:50 PM Kevin Risden  wrote:
> > I've noticed failures like this on PRs quite a few times when github
> > actions for `./gradlew check` pass no issue.
> >
> > https://github.com/apache/solr/actions/runs/4375627158/jobs/7656583100
> >
> > Setting up workspace (this could take some time)...
> > > Pulling container image...
> > > Finished pulling container image
> > > Switched to a new branch
> > > 'detached-57bb78866cc07e76f6d16cc0e08e1b6ef8701444'
> > > setting commitID to 90c4c50877c81fdf578c8ccf8f6976408eefaf27
> > > committer id not set in git config, hence setting it to Crave
> > > committer email not set in git config, hence setting it to
> > cr...@crave.io
> > > error: patch failed:
> > > solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:48
> > > error: solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:
> > > patch does not apply
> > > hint: Use 'git am --show-current-patch=diff' to see the failed patch
> > > Applying: SOLR-16681: Throw exception when attempting to replace
> > uniqueKey
> > > via fl in distributed request (#1384)
> > > Applying: SOLR-16676: Test improvements
> > > Applying: Avoid assertNotNull
> > > Applying: SOLR-16643: reRankOperator=multiply/replace options (#1431)
> > > Applying: SOLR-16676 Http2SolrClient loss of MDC context flaky test
> > changes
> > > Patch failed at 0005 SOLR-16676 Http2SolrClient loss of MDC context flaky
> > > test changes
> > > When you have resolved this problem, run "git am --continue".
> > > If you prefer to skip this patch, run "git am --skip" instead.
> > > To restore the original branch and stop patching, run "git am --abort".
> >
> >
> > The branch has main merged and is up to date so there should be no
> > conflicts. Based on the output its hard to tell why it failed to apply.
> >
> > Kevin Risden
> >
> >
> > On Thu, Feb 9, 2023 at 4:33 PM Jan Høydahl  wrote:
> >
> > > Looks like crave test (in github actions) is still somewhat unstable,
> > > sometimes failing to allocate resoources?
> > >
> > > Also, question regarding how to inspect crave failed tests. How to access
> > > the full log, it just says it is at file:///tmp/... but is there a way
> > for
> > > crave to generate a URL where one can access the test result output in
> > the
> > > cloud?
> > >
> > > Jan
> > >
> > > > 29. jan. 2023 kl. 20:32 skrev Yuvraaj Kelkar :
> > > >
> > > > Updated default command to
> > > > ./gradlew localSettings && ./gradlew --max-workers=`nproc`
> > > -Ptests.jvms=96 test
> > > >
> > > > Thanks,
> > > > -Uv
> > > >
> > > > On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya <
> > > ichattopadhy...@gmail.com> wrote:
> > > >> Thanks to Crave! Seems like there's Dual XEON 24 core processors
> > > >> available on the system.
> > > >>
> > > >> I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5
> > > min.
> > > >> ~/crave run -- './gradlew localSettings && ./gradlew
> > > >> --max-workers=`nproc` -Ptests.jvms=96 test'
> > > >> 
> > > >> The slowest tests (exceeding 500 ms) during this run:
> > > >> 94.39s BasicDistributedZkTest.test (:solr:core)
> > > >> 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
> > > >> 74.73s PeerSyncReplicationTest.test (:solr:core)
> > > >> 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
> > > >> (:solr:core)
> > > >> 59.15s ReplicationFactorTest.test (:solr:core)
> > > >> 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs)
> > > >> 57.89s TestDistributedSearch.test (:solr:core)
> > > >> 54.44s BasicDistributedZk2Test.test (:solr:core)
> > > >> 52.49s TestStressInPlaceUpdates.stressTest (:solr:core)
> > > >> 51.46s UnloadDistributedZkTest.test (:solr:core)
> > > >> The slowest suites (exceeding 1s) during this run:
> > > >> 188.60s CheckHdfsIndexTest (:solr:modules:hdfs)
> > > >> 184.41s TestRecoveryHdfs (:solr:modules:hdfs)
> > > >> 116.30s SchemaTest (:solr:solrj)
> > > >> 100.00s BasicDistributedZkTest (:solr:core)
> > > >> 94.33s StreamDecoratorTest (:solr:solrj-streaming)
> > > >> 88.09s TestTlogReplica (:solr:core)
> > > >> 83.08s ReindexCollectionTest (:solr:core)
> > > >> 79.25s TestCloudConsistency (:solr:core)
> > > >> 78.69s HdfsDirectoryTest (:solr:modules:hdfs)
> > > >> 76.33s MultiThreadedOCPTest (:solr:core)
> > > >>
> > > >> BUILD SUCCESSFUL in 5m 2s
> > > >> 153 actionable tasks: 55 executed, 98 up-to-date
> > > >>
> > > >>
> > 

Re: Running all tests via Crave.io

[Mikhail Khludnev]

Hello Kevin,
It happened to me too. I fixed it with rebasing pr branch.
$git pull --rebase origin main
$git push -f myfork PR-branch

On Thu, Mar 9, 2023 at 5:50 PM Kevin Risden  wrote:

> I've noticed failures like this on PRs quite a few times when github
> actions for `./gradlew check` pass no issue.
>
> https://github.com/apache/solr/actions/runs/4375627158/jobs/7656583100
>
> Setting up workspace (this could take some time)...
> > Pulling container image...
> > Finished pulling container image
> > Switched to a new branch
> > 'detached-57bb78866cc07e76f6d16cc0e08e1b6ef8701444'
> > setting commitID to 90c4c50877c81fdf578c8ccf8f6976408eefaf27
> > committer id not set in git config, hence setting it to Crave
> > committer email not set in git config, hence setting it to
> cr...@crave.io
> > error: patch failed:
> > solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:48
> > error: solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:
> > patch does not apply
> > hint: Use 'git am --show-current-patch=diff' to see the failed patch
> > Applying: SOLR-16681: Throw exception when attempting to replace
> uniqueKey
> > via fl in distributed request (#1384)
> > Applying: SOLR-16676: Test improvements
> > Applying: Avoid assertNotNull
> > Applying: SOLR-16643: reRankOperator=multiply/replace options (#1431)
> > Applying: SOLR-16676 Http2SolrClient loss of MDC context flaky test
> changes
> > Patch failed at 0005 SOLR-16676 Http2SolrClient loss of MDC context flaky
> > test changes
> > When you have resolved this problem, run "git am --continue".
> > If you prefer to skip this patch, run "git am --skip" instead.
> > To restore the original branch and stop patching, run "git am --abort".
>
>
> The branch has main merged and is up to date so there should be no
> conflicts. Based on the output its hard to tell why it failed to apply.
>
> Kevin Risden
>
>
> On Thu, Feb 9, 2023 at 4:33 PM Jan Høydahl  wrote:
>
> > Looks like crave test (in github actions) is still somewhat unstable,
> > sometimes failing to allocate resoources?
> >
> > Also, question regarding how to inspect crave failed tests. How to access
> > the full log, it just says it is at file:///tmp/... but is there a way
> for
> > crave to generate a URL where one can access the test result output in
> the
> > cloud?
> >
> > Jan
> >
> > > 29. jan. 2023 kl. 20:32 skrev Yuvraaj Kelkar :
> > >
> > > Updated default command to
> > > ./gradlew localSettings && ./gradlew --max-workers=`nproc`
> > -Ptests.jvms=96 test
> > >
> > > Thanks,
> > > -Uv
> > >
> > > On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya <
> > ichattopadhy...@gmail.com> wrote:
> > >> Thanks to Crave! Seems like there's Dual XEON 24 core processors
> > >> available on the system.
> > >>
> > >> I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5
> > min.
> > >> ~/crave run -- './gradlew localSettings && ./gradlew
> > >> --max-workers=`nproc` -Ptests.jvms=96 test'
> > >> 
> > >> The slowest tests (exceeding 500 ms) during this run:
> > >> 94.39s BasicDistributedZkTest.test (:solr:core)
> > >> 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
> > >> 74.73s PeerSyncReplicationTest.test (:solr:core)
> > >> 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
> > >> (:solr:core)
> > >> 59.15s ReplicationFactorTest.test (:solr:core)
> > >> 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs)
> > >> 57.89s TestDistributedSearch.test (:solr:core)
> > >> 54.44s BasicDistributedZk2Test.test (:solr:core)
> > >> 52.49s TestStressInPlaceUpdates.stressTest (:solr:core)
> > >> 51.46s UnloadDistributedZkTest.test (:solr:core)
> > >> The slowest suites (exceeding 1s) during this run:
> > >> 188.60s CheckHdfsIndexTest (:solr:modules:hdfs)
> > >> 184.41s TestRecoveryHdfs (:solr:modules:hdfs)
> > >> 116.30s SchemaTest (:solr:solrj)
> > >> 100.00s BasicDistributedZkTest (:solr:core)
> > >> 94.33s StreamDecoratorTest (:solr:solrj-streaming)
> > >> 88.09s TestTlogReplica (:solr:core)
> > >> 83.08s ReindexCollectionTest (:solr:core)
> > >> 79.25s TestCloudConsistency (:solr:core)
> > >> 78.69s HdfsDirectoryTest (:solr:modules:hdfs)
> > >> 76.33s MultiThreadedOCPTest (:solr:core)
> > >>
> > >> BUILD SUCCESSFUL in 5m 2s
> > >> 153 actionable tasks: 55 executed, 98 up-to-date
> > >>
> > >>
> 
> > >> Build Successful
> > >>
> > >> Total time: 5m25.5s
> > >>
> 
> > >>
> > >> On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh
> > >>  wrote:
> > >>>
> > >>> Fantastic!
> > >>>
> >  On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar 
> > wrote:
> > 
> >  Hurray!
> > 
> >  On Fri, Jan 27, 2023, 19:27 David Smiley 
> wrote:
> > 
> > > The PR is merged. Henceforth, all java based tests will run on PRs.
> > > Also "crave run" now has the defaults so you can run that at your
> > termin

Re: Running all tests via Crave.io

[Kevin Risden]

I've noticed failures like this on PRs quite a few times when github
actions for `./gradlew check` pass no issue.

https://github.com/apache/solr/actions/runs/4375627158/jobs/7656583100

Setting up workspace (this could take some time)...
> Pulling container image...
> Finished pulling container image
> Switched to a new branch
> 'detached-57bb78866cc07e76f6d16cc0e08e1b6ef8701444'
> setting commitID to 90c4c50877c81fdf578c8ccf8f6976408eefaf27
> committer id not set in git config, hence setting it to Crave
> committer email not set in git config, hence setting it to cr...@crave.io
> error: patch failed:
> solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:48
> error: solr/core/src/test/org/apache/solr/handler/TestHttpRequestId.java:
> patch does not apply
> hint: Use 'git am --show-current-patch=diff' to see the failed patch
> Applying: SOLR-16681: Throw exception when attempting to replace uniqueKey
> via fl in distributed request (#1384)
> Applying: SOLR-16676: Test improvements
> Applying: Avoid assertNotNull
> Applying: SOLR-16643: reRankOperator=multiply/replace options (#1431)
> Applying: SOLR-16676 Http2SolrClient loss of MDC context flaky test changes
> Patch failed at 0005 SOLR-16676 Http2SolrClient loss of MDC context flaky
> test changes
> When you have resolved this problem, run "git am --continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".


The branch has main merged and is up to date so there should be no
conflicts. Based on the output its hard to tell why it failed to apply.

Kevin Risden


On Thu, Feb 9, 2023 at 4:33 PM Jan Høydahl  wrote:

> Looks like crave test (in github actions) is still somewhat unstable,
> sometimes failing to allocate resoources?
>
> Also, question regarding how to inspect crave failed tests. How to access
> the full log, it just says it is at file:///tmp/... but is there a way for
> crave to generate a URL where one can access the test result output in the
> cloud?
>
> Jan
>
> > 29. jan. 2023 kl. 20:32 skrev Yuvraaj Kelkar :
> >
> > Updated default command to
> > ./gradlew localSettings && ./gradlew --max-workers=`nproc`
> -Ptests.jvms=96 test
> >
> > Thanks,
> > -Uv
> >
> > On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
> >> Thanks to Crave! Seems like there's Dual XEON 24 core processors
> >> available on the system.
> >>
> >> I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5
> min.
> >> ~/crave run -- './gradlew localSettings && ./gradlew
> >> --max-workers=`nproc` -Ptests.jvms=96 test'
> >> 
> >> The slowest tests (exceeding 500 ms) during this run:
> >> 94.39s BasicDistributedZkTest.test (:solr:core)
> >> 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
> >> 74.73s PeerSyncReplicationTest.test (:solr:core)
> >> 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
> >> (:solr:core)
> >> 59.15s ReplicationFactorTest.test (:solr:core)
> >> 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs)
> >> 57.89s TestDistributedSearch.test (:solr:core)
> >> 54.44s BasicDistributedZk2Test.test (:solr:core)
> >> 52.49s TestStressInPlaceUpdates.stressTest (:solr:core)
> >> 51.46s UnloadDistributedZkTest.test (:solr:core)
> >> The slowest suites (exceeding 1s) during this run:
> >> 188.60s CheckHdfsIndexTest (:solr:modules:hdfs)
> >> 184.41s TestRecoveryHdfs (:solr:modules:hdfs)
> >> 116.30s SchemaTest (:solr:solrj)
> >> 100.00s BasicDistributedZkTest (:solr:core)
> >> 94.33s StreamDecoratorTest (:solr:solrj-streaming)
> >> 88.09s TestTlogReplica (:solr:core)
> >> 83.08s ReindexCollectionTest (:solr:core)
> >> 79.25s TestCloudConsistency (:solr:core)
> >> 78.69s HdfsDirectoryTest (:solr:modules:hdfs)
> >> 76.33s MultiThreadedOCPTest (:solr:core)
> >>
> >> BUILD SUCCESSFUL in 5m 2s
> >> 153 actionable tasks: 55 executed, 98 up-to-date
> >>
> >> 
> >> Build Successful
> >>
> >> Total time: 5m25.5s
> >> 
> >>
> >> On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh
> >>  wrote:
> >>>
> >>> Fantastic!
> >>>
>  On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar 
> wrote:
> 
>  Hurray!
> 
>  On Fri, Jan 27, 2023, 19:27 David Smiley  wrote:
> 
> > The PR is merged. Henceforth, all java based tests will run on PRs.
> > Also "crave run" now has the defaults so you can run that at your
> terminal
> > without adding all the extra args :-)
> >
> > I did an experiment[1] wondering how long it would take a typical
> GitHub
> > Action machine to run all these Solr tests. The answer is 96
> minutes. No
> > failures / timeouts; it really took that long.
> > [1]
> https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search

Re: Running all tests via Crave.io

[Jan Høydahl]

Looks like crave test (in github actions) is still somewhat unstable, sometimes 
failing to allocate resoources?

Also, question regarding how to inspect crave failed tests. How to access the 
full log, it just says it is at file:///tmp/... but is there a way for crave to 
generate a URL where one can access the test result output in the cloud?

Jan

> 29. jan. 2023 kl. 20:32 skrev Yuvraaj Kelkar :
> 
> Updated default command to
> ./gradlew localSettings && ./gradlew --max-workers=`nproc` -Ptests.jvms=96 
> test
> 
> Thanks,
> -Uv
> 
> On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya  
> wrote:
>> Thanks to Crave! Seems like there's Dual XEON 24 core processors
>> available on the system.
>> 
>> I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5 min.
>> ~/crave run -- './gradlew localSettings && ./gradlew
>> --max-workers=`nproc` -Ptests.jvms=96 test'
>> 
>> The slowest tests (exceeding 500 ms) during this run:
>> 94.39s BasicDistributedZkTest.test (:solr:core)
>> 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
>> 74.73s PeerSyncReplicationTest.test (:solr:core)
>> 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
>> (:solr:core)
>> 59.15s ReplicationFactorTest.test (:solr:core)
>> 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs)
>> 57.89s TestDistributedSearch.test (:solr:core)
>> 54.44s BasicDistributedZk2Test.test (:solr:core)
>> 52.49s TestStressInPlaceUpdates.stressTest (:solr:core)
>> 51.46s UnloadDistributedZkTest.test (:solr:core)
>> The slowest suites (exceeding 1s) during this run:
>> 188.60s CheckHdfsIndexTest (:solr:modules:hdfs)
>> 184.41s TestRecoveryHdfs (:solr:modules:hdfs)
>> 116.30s SchemaTest (:solr:solrj)
>> 100.00s BasicDistributedZkTest (:solr:core)
>> 94.33s StreamDecoratorTest (:solr:solrj-streaming)
>> 88.09s TestTlogReplica (:solr:core)
>> 83.08s ReindexCollectionTest (:solr:core)
>> 79.25s TestCloudConsistency (:solr:core)
>> 78.69s HdfsDirectoryTest (:solr:modules:hdfs)
>> 76.33s MultiThreadedOCPTest (:solr:core)
>> 
>> BUILD SUCCESSFUL in 5m 2s
>> 153 actionable tasks: 55 executed, 98 up-to-date
>> 
>> 
>> Build Successful
>> 
>> Total time: 5m25.5s
>> 
>> 
>> On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh
>>  wrote:
>>> 
>>> Fantastic!
>>> 
 On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar  wrote:
 
 Hurray!
 
 On Fri, Jan 27, 2023, 19:27 David Smiley  wrote:
 
> The PR is merged. Henceforth, all java based tests will run on PRs.
> Also "crave run" now has the defaults so you can run that at your terminal
> without adding all the extra args :-)
> 
> I did an experiment[1] wondering how long it would take a typical GitHub
> Action machine to run all these Solr tests. The answer is 96 minutes. No
> failures / timeouts; it really took that long.
> [1] https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956
> 
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
> 
> 
> On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:
> 
>> We haven't been running all our tests in GitHub Actions (i.e. PR
>> validation) because it was too time consuming to do so. I don't recall
> how
>> slow it was when someone last tried; it's probably better now but still
>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches
>> SolrJ.
>> 
>> There's now a PR here to introduce a new GHA that builds on Crave.io on a
>> beefy machine: https://github.com/apache/solr/pull/1303 The PR
>> validation took 11 minutes which is similar to the amount of time it
> took a
>> GHA to just do precommit checks -- 10 minutes :-)
>> I think we can remove the SolrJ specific GHA as it'll be redundant.
>> 
>> Furthermore, anyone can use this to run tests from the convenience of
> your
>> laptop at the CLI while you are in the middle of any change (doesn't
> matter
>> what you have committed or not, pushed or not). To do so, run: crave run
>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
>> -Ptests.jvms=48 test'
>> 
>> Yeah that's long. There is a discussion in JIRA underway that may lead
> to
>> eliminating the "localSettings" step if, for example, it's moved to a
> bash
>> script executed by the gradle wrapper (my proposal). I should also be
> able
>> to configure crave with a default run configuration with this baked in.
>> I'll post an update when I'm able to do that.
>> 
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>> 
> 
>>> 
>>> ___
>>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
>>> http://

Re: Running all tests via Crave.io

[Yuvraaj Kelkar]

Updated default command to
./gradlew localSettings && ./gradlew --max-workers=`nproc` -Ptests.jvms=96 test

Thanks,
-Uv

On Jan 29 2023, at 5:00 am, Ishan Chattopadhyaya  
wrote:
> Thanks to Crave! Seems like there's Dual XEON 24 core processors
> available on the system.
>
> I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5 min.
> ~/crave run -- './gradlew localSettings && ./gradlew
> --max-workers=`nproc` -Ptests.jvms=96 test'
> 
> The slowest tests (exceeding 500 ms) during this run:
> 94.39s BasicDistributedZkTest.test (:solr:core)
> 90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
> 74.73s PeerSyncReplicationTest.test (:solr:core)
> 71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
> (:solr:core)
> 59.15s ReplicationFactorTest.test (:solr:core)
> 58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs)
> 57.89s TestDistributedSearch.test (:solr:core)
> 54.44s BasicDistributedZk2Test.test (:solr:core)
> 52.49s TestStressInPlaceUpdates.stressTest (:solr:core)
> 51.46s UnloadDistributedZkTest.test (:solr:core)
> The slowest suites (exceeding 1s) during this run:
> 188.60s CheckHdfsIndexTest (:solr:modules:hdfs)
> 184.41s TestRecoveryHdfs (:solr:modules:hdfs)
> 116.30s SchemaTest (:solr:solrj)
> 100.00s BasicDistributedZkTest (:solr:core)
> 94.33s StreamDecoratorTest (:solr:solrj-streaming)
> 88.09s TestTlogReplica (:solr:core)
> 83.08s ReindexCollectionTest (:solr:core)
> 79.25s TestCloudConsistency (:solr:core)
> 78.69s HdfsDirectoryTest (:solr:modules:hdfs)
> 76.33s MultiThreadedOCPTest (:solr:core)
>
> BUILD SUCCESSFUL in 5m 2s
> 153 actionable tasks: 55 executed, 98 up-to-date
>
> 
> Build Successful
>
> Total time: 5m25.5s
> 
>
> On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh
>  wrote:
> >
> > Fantastic!
> >
> > > On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar  wrote:
> > >
> > > Hurray!
> > >
> > > On Fri, Jan 27, 2023, 19:27 David Smiley  wrote:
> > >
> > >> The PR is merged. Henceforth, all java based tests will run on PRs.
> > >> Also "crave run" now has the defaults so you can run that at your 
> > >> terminal
> > >> without adding all the extra args :-)
> > >>
> > >> I did an experiment[1] wondering how long it would take a typical GitHub
> > >> Action machine to run all these Solr tests. The answer is 96 minutes. No
> > >> failures / timeouts; it really took that long.
> > >> [1] 
> > >> https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956
> > >>
> > >> ~ David Smiley
> > >> Apache Lucene/Solr Search Developer
> > >> http://www.linkedin.com/in/davidwsmiley
> > >>
> > >>
> > >> On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:
> > >>
> > >>> We haven't been running all our tests in GitHub Actions (i.e. PR
> > >>> validation) because it was too time consuming to do so. I don't recall
> > >> how
> > >>> slow it was when someone last tried; it's probably better now but still
> > >>> slow. To make up for this, there is a GHA only for SolrJ if a PR touches
> > >>> SolrJ.
> > >>>
> > >>> There's now a PR here to introduce a new GHA that builds on Crave.io on 
> > >>> a
> > >>> beefy machine: https://github.com/apache/solr/pull/1303 The PR
> > >>> validation took 11 minutes which is similar to the amount of time it
> > >> took a
> > >>> GHA to just do precommit checks -- 10 minutes :-)
> > >>> I think we can remove the SolrJ specific GHA as it'll be redundant.
> > >>>
> > >>> Furthermore, anyone can use this to run tests from the convenience of
> > >> your
> > >>> laptop at the CLI while you are in the middle of any change (doesn't
> > >> matter
> > >>> what you have committed or not, pushed or not). To do so, run: crave run
> > >>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > >>> -Ptests.jvms=48 test'
> > >>>
> > >>> Yeah that's long. There is a discussion in JIRA underway that may lead
> > >> to
> > >>> eliminating the "localSettings" step if, for example, it's moved to a
> > >> bash
> > >>> script executed by the gradle wrapper (my proposal). I should also be
> > >> able
> > >>> to configure crave with a default run configuration with this baked in.
> > >>> I'll post an update when I'm able to do that.
> > >>>
> > >>> ~ David Smiley
> > >>> Apache Lucene/Solr Search Developer
> > >>> http://www.linkedin.com/in/davidwsmiley
> > >>>
> > >>
> >
> > ___
> > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> > http://www.opensourceconnections.com 
> >  | My Free/Busy 
> > 
> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 
> > 
> > This e-mail and all contents, including attachments, is considered to be 
> > Company Confiden

Re: Running all tests via Crave.io

[Ishan Chattopadhyaya]

Thanks to Crave! Seems like there's Dual XEON 24 core processors
available on the system.

I pushed the parallelism to 96, and times reduced from 8.5 min to ~5.5 min.

~/crave run -- './gradlew localSettings && ./gradlew
--max-workers=`nproc` -Ptests.jvms=96 test'

The slowest tests (exceeding 500 ms) during this run:
  94.39s BasicDistributedZkTest.test (:solr:core)
  90.12s CheckHdfsIndexTest.testDeletedDocs (:solr:modules:hdfs)
  74.73s PeerSyncReplicationTest.test (:solr:core)
  71.84s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection
(:solr:core)
  59.15s ReplicationFactorTest.test (:solr:core)
  58.82s HdfsDirectoryTest.testRandomAccessWrites (:solr:modules:hdfs)
  57.89s TestDistributedSearch.test (:solr:core)
  54.44s BasicDistributedZk2Test.test (:solr:core)
  52.49s TestStressInPlaceUpdates.stressTest (:solr:core)
  51.46s UnloadDistributedZkTest.test (:solr:core)
The slowest suites (exceeding 1s) during this run:
  188.60s CheckHdfsIndexTest (:solr:modules:hdfs)
  184.41s TestRecoveryHdfs (:solr:modules:hdfs)
  116.30s SchemaTest (:solr:solrj)
  100.00s BasicDistributedZkTest (:solr:core)
  94.33s StreamDecoratorTest (:solr:solrj-streaming)
  88.09s TestTlogReplica (:solr:core)
  83.08s ReindexCollectionTest (:solr:core)
  79.25s TestCloudConsistency (:solr:core)
  78.69s HdfsDirectoryTest (:solr:modules:hdfs)
  76.33s MultiThreadedOCPTest (:solr:core)

BUILD SUCCESSFUL in 5m 2s
153 actionable tasks: 55 executed, 98 up-to-date


Build Successful

Total time: 5m25.5s


On Sat, Jan 28, 2023 at 6:06 PM Eric Pugh
 wrote:
>
> Fantastic!
>
> > On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar  wrote:
> >
> > Hurray!
> >
> > On Fri, Jan 27, 2023, 19:27 David Smiley  wrote:
> >
> >> The PR is merged.  Henceforth, all java based tests will run on PRs.
> >> Also "crave run" now has the defaults so you can run that at your terminal
> >> without adding all the extra args :-)
> >>
> >> I did an experiment[1] wondering how long it would take a typical GitHub
> >> Action machine to run all these Solr tests.  The answer is 96 minutes.  No
> >> failures / timeouts; it really took that long.
> >> [1] https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956
> >>
> >> ~ David Smiley
> >> Apache Lucene/Solr Search Developer
> >> http://www.linkedin.com/in/davidwsmiley
> >>
> >>
> >> On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:
> >>
> >>> We haven't been running all our tests in GitHub Actions (i.e. PR
> >>> validation) because it was too time consuming to do so.  I don't recall
> >> how
> >>> slow it was when someone last tried; it's probably better now but still
> >>> slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
> >>> SolrJ.
> >>>
> >>> There's now a PR here to introduce a new GHA that builds on Crave.io on a
> >>> beefy machine: https://github.com/apache/solr/pull/1303  The PR
> >>> validation took 11 minutes which is similar to the amount of time it
> >> took a
> >>> GHA to just do precommit checks -- 10 minutes :-)
> >>> I think we can remove the SolrJ specific GHA as it'll be redundant.
> >>>
> >>> Furthermore, anyone can use this to run tests from the convenience of
> >> your
> >>> laptop at the CLI while you are in the middle of any change (doesn't
> >> matter
> >>> what you have committed or not, pushed or not).  To do so, run: crave run
> >>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> >>> -Ptests.jvms=48 test'
> >>>
> >>> Yeah that's long.  There is a discussion in JIRA underway that may lead
> >> to
> >>> eliminating the "localSettings" step if, for example, it's moved to a
> >> bash
> >>> script executed by the gradle wrapper (my proposal).  I should also be
> >> able
> >>> to configure crave with a default run configuration with this baked in.
> >>> I'll post an update when I'm able to do that.
> >>>
> >>> ~ David Smiley
> >>> Apache Lucene/Solr Search Developer
> >>> http://www.linkedin.com/in/davidwsmiley
> >>>
> >>
>
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> http://www.opensourceconnections.com  
> | My Free/Busy 
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 
> 
> This e-mail and all contents, including attachments, is considered to be 
> Company Confidential unless explicitly stated otherwise, regardless of 
> whether attachments are marked as such.
>

-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Running all tests via Crave.io

[Eric Pugh]

Fantastic!   

> On Jan 27, 2023, at 10:36 PM, Yuvraaj Kelkar  wrote:
> 
> Hurray!
> 
> On Fri, Jan 27, 2023, 19:27 David Smiley  wrote:
> 
>> The PR is merged.  Henceforth, all java based tests will run on PRs.
>> Also "crave run" now has the defaults so you can run that at your terminal
>> without adding all the extra args :-)
>> 
>> I did an experiment[1] wondering how long it would take a typical GitHub
>> Action machine to run all these Solr tests.  The answer is 96 minutes.  No
>> failures / timeouts; it really took that long.
>> [1] https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956
>> 
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>> 
>> 
>> On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:
>> 
>>> We haven't been running all our tests in GitHub Actions (i.e. PR
>>> validation) because it was too time consuming to do so.  I don't recall
>> how
>>> slow it was when someone last tried; it's probably better now but still
>>> slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
>>> SolrJ.
>>> 
>>> There's now a PR here to introduce a new GHA that builds on Crave.io on a
>>> beefy machine: https://github.com/apache/solr/pull/1303  The PR
>>> validation took 11 minutes which is similar to the amount of time it
>> took a
>>> GHA to just do precommit checks -- 10 minutes :-)
>>> I think we can remove the SolrJ specific GHA as it'll be redundant.
>>> 
>>> Furthermore, anyone can use this to run tests from the convenience of
>> your
>>> laptop at the CLI while you are in the middle of any change (doesn't
>> matter
>>> what you have committed or not, pushed or not).  To do so, run: crave run
>>> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
>>> -Ptests.jvms=48 test'
>>> 
>>> Yeah that's long.  There is a discussion in JIRA underway that may lead
>> to
>>> eliminating the "localSettings" step if, for example, it's moved to a
>> bash
>>> script executed by the gradle wrapper (my proposal).  I should also be
>> able
>>> to configure crave with a default run configuration with this baked in.
>>> I'll post an update when I'm able to do that.
>>> 
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley
>>> 
>> 

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Re: Running all tests via Crave.io

[Yuvraaj Kelkar]

Hurray!

On Fri, Jan 27, 2023, 19:27 David Smiley  wrote:

> The PR is merged.  Henceforth, all java based tests will run on PRs.
> Also "crave run" now has the defaults so you can run that at your terminal
> without adding all the extra args :-)
>
> I did an experiment[1] wondering how long it would take a typical GitHub
> Action machine to run all these Solr tests.  The answer is 96 minutes.  No
> failures / timeouts; it really took that long.
> [1] https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:
>
> > We haven't been running all our tests in GitHub Actions (i.e. PR
> > validation) because it was too time consuming to do so.  I don't recall
> how
> > slow it was when someone last tried; it's probably better now but still
> > slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
> > SolrJ.
> >
> > There's now a PR here to introduce a new GHA that builds on Crave.io on a
> > beefy machine: https://github.com/apache/solr/pull/1303  The PR
> > validation took 11 minutes which is similar to the amount of time it
> took a
> > GHA to just do precommit checks -- 10 minutes :-)
> > I think we can remove the SolrJ specific GHA as it'll be redundant.
> >
> > Furthermore, anyone can use this to run tests from the convenience of
> your
> > laptop at the CLI while you are in the middle of any change (doesn't
> matter
> > what you have committed or not, pushed or not).  To do so, run: crave run
> > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > -Ptests.jvms=48 test'
> >
> > Yeah that's long.  There is a discussion in JIRA underway that may lead
> to
> > eliminating the "localSettings" step if, for example, it's moved to a
> bash
> > script executed by the gradle wrapper (my proposal).  I should also be
> able
> > to configure crave with a default run configuration with this baked in.
> > I'll post an update when I'm able to do that.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
>

Re: Running all tests via Crave.io

[David Smiley]

The PR is merged.  Henceforth, all java based tests will run on PRs.
Also "crave run" now has the defaults so you can run that at your terminal
without adding all the extra args :-)

I did an experiment[1] wondering how long it would take a typical GitHub
Action machine to run all these Solr tests.  The answer is 96 minutes.  No
failures / timeouts; it really took that long.
[1] https://github.com/apache/solr/actions/runs/4028015923/jobs/6924417956

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:

> We haven't been running all our tests in GitHub Actions (i.e. PR
> validation) because it was too time consuming to do so.  I don't recall how
> slow it was when someone last tried; it's probably better now but still
> slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
> SolrJ.
>
> There's now a PR here to introduce a new GHA that builds on Crave.io on a
> beefy machine: https://github.com/apache/solr/pull/1303  The PR
> validation took 11 minutes which is similar to the amount of time it took a
> GHA to just do precommit checks -- 10 minutes :-)
> I think we can remove the SolrJ specific GHA as it'll be redundant.
>
> Furthermore, anyone can use this to run tests from the convenience of your
> laptop at the CLI while you are in the middle of any change (doesn't matter
> what you have committed or not, pushed or not).  To do so, run: crave run
> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> -Ptests.jvms=48 test'
>
> Yeah that's long.  There is a discussion in JIRA underway that may lead to
> eliminating the "localSettings" step if, for example, it's moved to a bash
> script executed by the gradle wrapper (my proposal).  I should also be able
> to configure crave with a default run configuration with this baked in.
> I'll post an update when I'm able to do that.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

Re: Running all tests via Crave.io

[Mike Drob]

@Ishan - not necessarily, a committer has to explicitly approve running the
GHA checks for first time contributors. So a random person with a drive-by
PR won't be automatically handed the keys to our infra. I believe we can
set that to needing approval for each run by a non-committer, or maybe
that's already set, somebody would need to dig through GH configs and
verify (or experiment).

I think that would still run into issues with my second case. Even if we
allow 'gradlew test' in the list of allowed commands (which I think we
would have to), what's stopping a user from creating a new test (or
modifying an existing one) to do malicious things and then execute it via
the test runner. It doesn't matter if they're launching bitcoin miners on
their own machine - using our tests to do that would be a very indirect and
inefficient way to do it. But giving people access to unfettered compute
resources suddenly makes for an enticing target.

I'm not sure if I'm being clear enough with my description, so let me try
an example. `gradlew test` will run all tests. The attacker creates a new
test "DoBadThingsTest" that will connect out to the internet and do the
previously described bad things. They launch crave run from their machine
with the changed code, and... does this upload their current code state to
crave to run the tests?

This is probably possible to do safely with sufficient isolation - gradle
needs access to a specific set of known hosts to download dependencies, and
the tests need network connectivity to talk to the multiple Solr servers
that we launch during testing, but otherwise I don't think there's a need
for external traffic. This is hard to do correctly and completely, but
would essentially limits the threat model to a DoS.

Anyway, it's not my place to tell Crave how to run their business, and I
don't mean to turn down their generous contribution with a heavy-handed
security audit. It's a very nice thing that they are offering and I think
we should take advantage of it while it is available.

Mike

On Thu, Jan 26, 2023 at 9:18 AM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Though, if a malicious user creates a PR that executes harmful code, that
> PR will also get executed (via "gradlew test"), right?
>
> On Thu, 26 Jan, 2023, 8:36 pm Yuvraaj Kelkar,  wrote:
>
> > Hm. I see your point. The first solution I thought of is a bit blunt, but
> > it will work: We disable command line arguments when using ZeroConf. Only
> > preset commands are allowed.
> > Also, to allow the Solr GHA to run whatever commands needed not just for
> > now, but also for any future changes, we set up regular (not ZeroConf)
> > access for the GHA runner.
> > This way GHA always has the ability to run any commands - but it is
> > strictly controlled by review requests.
> > Regular developers only get to use the commands that are approved by the
> > Solr community leaders.
> >
> > Would that work?
> > Thanks,
> > -Uv
> >
> > On Jan 26 2023, at 8:09 am, Mike Drob  wrote:
> > > Having massive infrastructure to run PRs is pretty cool.
> > >
> > > I'm worried about letting arbitrary people run code on these
> > > machines though - a single 'crave run -- mine_bitcoin.exe' would ruin
> the
> > > system for everybody, or it's not hard to imagine a slightly more
> > indirect
> > > case where an attacker adds a test that launches an undesirable process
> > and
> > > runs crave. What safeguards exist to protect against this? At least
> with
> > > GHA we have to approve non-committers tests to run, but opening it up
> to
> > > local command line access sounds very broad.
> > >
> > > Mike
> > > On Wed, Jan 25, 2023 at 10:34 PM Ishan Chattopadhyaya <
> > > ichattopadhy...@gmail.com> wrote:
> > >
> > > > This is very cool. Thanks for working on this, David. Can multiple
> > > > developers execute their tests at the same time?
> > > >
> > > > On Thu, 26 Jan, 2023, 5:07 am Noble Paul, 
> > wrote:
> > > >
> > > > > This is interesting.
> > > > >
> > > > > So, if the PR is merged , we will have the full test running on
> > crave.io
> > > > > for every PR raised?
> > > > >
> > > > > On Thu, Jan 26, 2023 at 9:22 AM David Smiley 
> > wrote:
> > > > >
> > > > > > We haven't been running all our tests in GitHub Actions (i.e. PR
> > > > > > validation) because it was too time consuming to do so. I don't
> > recall
> > > > > how
> > > > > > slow it was when someone last tried; it's probably better now but
> > still
> > > > > > slow. To make up for this, there is a GHA only for SolrJ if a PR
> > > > touches
> > > > > > SolrJ.
> > > > > >
> > > > > > There's now a PR here to introduce a new GHA that builds on
> > Crave.io
> > > > on a
> > > > > > beefy machine: https://github.com/apache/solr/pull/1303 The PR
> > > > > validation
> > > > > > took 11 minutes which is similar to the amount of time it took a
> > GHA to
> > > > > > just do precommit checks -- 10 minutes :-)
> > > > > > I think we can remove the SolrJ specific GHA as it'll be
>

Re: Running all tests via Crave.io

[Houston Putman]

Non-committers’ PRs do not have Github Actions run automatically. A
committer has to press a button to allow the tests to run. So unless a
committer is malicious, there shouldnt be a problem there.

On Thu, Jan 26, 2023 at 10:18 AM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Though, if a malicious user creates a PR that executes harmful code, that
> PR will also get executed (via "gradlew test"), right?
>
> On Thu, 26 Jan, 2023, 8:36 pm Yuvraaj Kelkar,  wrote:
>
> > Hm. I see your point. The first solution I thought of is a bit blunt, but
> > it will work: We disable command line arguments when using ZeroConf. Only
> > preset commands are allowed.
> > Also, to allow the Solr GHA to run whatever commands needed not just for
> > now, but also for any future changes, we set up regular (not ZeroConf)
> > access for the GHA runner.
> > This way GHA always has the ability to run any commands - but it is
> > strictly controlled by review requests.
> > Regular developers only get to use the commands that are approved by the
> > Solr community leaders.
> >
> > Would that work?
> > Thanks,
> > -Uv
> >
> > On Jan 26 2023, at 8:09 am, Mike Drob  wrote:
> > > Having massive infrastructure to run PRs is pretty cool.
> > >
> > > I'm worried about letting arbitrary people run code on these
> > > machines though - a single 'crave run -- mine_bitcoin.exe' would ruin
> the
> > > system for everybody, or it's not hard to imagine a slightly more
> > indirect
> > > case where an attacker adds a test that launches an undesirable process
> > and
> > > runs crave. What safeguards exist to protect against this? At least
> with
> > > GHA we have to approve non-committers tests to run, but opening it up
> to
> > > local command line access sounds very broad.
> > >
> > > Mike
> > > On Wed, Jan 25, 2023 at 10:34 PM Ishan Chattopadhyaya <
> > > ichattopadhy...@gmail.com> wrote:
> > >
> > > > This is very cool. Thanks for working on this, David. Can multiple
> > > > developers execute their tests at the same time?
> > > >
> > > > On Thu, 26 Jan, 2023, 5:07 am Noble Paul, 
> > wrote:
> > > >
> > > > > This is interesting.
> > > > >
> > > > > So, if the PR is merged , we will have the full test running on
> > crave.io
> > > > > for every PR raised?
> > > > >
> > > > > On Thu, Jan 26, 2023 at 9:22 AM David Smiley 
> > wrote:
> > > > >
> > > > > > We haven't been running all our tests in GitHub Actions (i.e. PR
> > > > > > validation) because it was too time consuming to do so. I don't
> > recall
> > > > > how
> > > > > > slow it was when someone last tried; it's probably better now but
> > still
> > > > > > slow. To make up for this, there is a GHA only for SolrJ if a PR
> > > > touches
> > > > > > SolrJ.
> > > > > >
> > > > > > There's now a PR here to introduce a new GHA that builds on
> > Crave.io
> > > > on a
> > > > > > beefy machine: https://github.com/apache/solr/pull/1303 The PR
> > > > > validation
> > > > > > took 11 minutes which is similar to the amount of time it took a
> > GHA to
> > > > > > just do precommit checks -- 10 minutes :-)
> > > > > > I think we can remove the SolrJ specific GHA as it'll be
> redundant.
> > > > > >
> > > > > > Furthermore, anyone can use this to run tests from the
> convenience
> > of
> > > > > your
> > > > > > laptop at the CLI while you are in the middle of any change
> > (doesn't
> > > > > matter
> > > > > > what you have committed or not, pushed or not). To do so, run:
> > crave
> > > > run
> > > > > > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > > > > > -Ptests.jvms=48 test'
> > > > > >
> > > > > > Yeah that's long. There is a discussion in JIRA underway that may
> > lead
> > > > > to
> > > > > > eliminating the "localSettings" step if, for example, it's moved
> > to a
> > > > > bash
> > > > > > script executed by the gradle wrapper (my proposal). I should
> also
> > be
> > > > > able
> > > > > > to configure crave with a default run configuration with this
> > baked in.
> > > > > > I'll post an update when I'm able to do that.
> > > > > >
> > > > > > ~ David Smiley
> > > > > > Apache Lucene/Solr Search Developer
> > > > > > http://www.linkedin.com/in/davidwsmiley
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -
> > > > > Noble Paul
> > > > >
> > > >
> > >
> >
> >
>

Re: Running all tests via Crave.io

[Ishan Chattopadhyaya]

Though, if a malicious user creates a PR that executes harmful code, that
PR will also get executed (via "gradlew test"), right?

On Thu, 26 Jan, 2023, 8:36 pm Yuvraaj Kelkar,  wrote:

> Hm. I see your point. The first solution I thought of is a bit blunt, but
> it will work: We disable command line arguments when using ZeroConf. Only
> preset commands are allowed.
> Also, to allow the Solr GHA to run whatever commands needed not just for
> now, but also for any future changes, we set up regular (not ZeroConf)
> access for the GHA runner.
> This way GHA always has the ability to run any commands - but it is
> strictly controlled by review requests.
> Regular developers only get to use the commands that are approved by the
> Solr community leaders.
>
> Would that work?
> Thanks,
> -Uv
>
> On Jan 26 2023, at 8:09 am, Mike Drob  wrote:
> > Having massive infrastructure to run PRs is pretty cool.
> >
> > I'm worried about letting arbitrary people run code on these
> > machines though - a single 'crave run -- mine_bitcoin.exe' would ruin the
> > system for everybody, or it's not hard to imagine a slightly more
> indirect
> > case where an attacker adds a test that launches an undesirable process
> and
> > runs crave. What safeguards exist to protect against this? At least with
> > GHA we have to approve non-committers tests to run, but opening it up to
> > local command line access sounds very broad.
> >
> > Mike
> > On Wed, Jan 25, 2023 at 10:34 PM Ishan Chattopadhyaya <
> > ichattopadhy...@gmail.com> wrote:
> >
> > > This is very cool. Thanks for working on this, David. Can multiple
> > > developers execute their tests at the same time?
> > >
> > > On Thu, 26 Jan, 2023, 5:07 am Noble Paul, 
> wrote:
> > >
> > > > This is interesting.
> > > >
> > > > So, if the PR is merged , we will have the full test running on
> crave.io
> > > > for every PR raised?
> > > >
> > > > On Thu, Jan 26, 2023 at 9:22 AM David Smiley 
> wrote:
> > > >
> > > > > We haven't been running all our tests in GitHub Actions (i.e. PR
> > > > > validation) because it was too time consuming to do so. I don't
> recall
> > > > how
> > > > > slow it was when someone last tried; it's probably better now but
> still
> > > > > slow. To make up for this, there is a GHA only for SolrJ if a PR
> > > touches
> > > > > SolrJ.
> > > > >
> > > > > There's now a PR here to introduce a new GHA that builds on
> Crave.io
> > > on a
> > > > > beefy machine: https://github.com/apache/solr/pull/1303 The PR
> > > > validation
> > > > > took 11 minutes which is similar to the amount of time it took a
> GHA to
> > > > > just do precommit checks -- 10 minutes :-)
> > > > > I think we can remove the SolrJ specific GHA as it'll be redundant.
> > > > >
> > > > > Furthermore, anyone can use this to run tests from the convenience
> of
> > > > your
> > > > > laptop at the CLI while you are in the middle of any change
> (doesn't
> > > > matter
> > > > > what you have committed or not, pushed or not). To do so, run:
> crave
> > > run
> > > > > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > > > > -Ptests.jvms=48 test'
> > > > >
> > > > > Yeah that's long. There is a discussion in JIRA underway that may
> lead
> > > > to
> > > > > eliminating the "localSettings" step if, for example, it's moved
> to a
> > > > bash
> > > > > script executed by the gradle wrapper (my proposal). I should also
> be
> > > > able
> > > > > to configure crave with a default run configuration with this
> baked in.
> > > > > I'll post an update when I'm able to do that.
> > > > >
> > > > > ~ David Smiley
> > > > > Apache Lucene/Solr Search Developer
> > > > > http://www.linkedin.com/in/davidwsmiley
> > > > >
> > > >
> > > >
> > > > --
> > > > -
> > > > Noble Paul
> > > >
> > >
> >
>
>

Re: Running all tests via Crave.io

[Yuvraaj Kelkar]

Hm. I see your point. The first solution I thought of is a bit blunt, but it 
will work: We disable command line arguments when using ZeroConf. Only preset 
commands are allowed.
Also, to allow the Solr GHA to run whatever commands needed not just for now, 
but also for any future changes, we set up regular (not ZeroConf) access for 
the GHA runner.
This way GHA always has the ability to run any commands - but it is strictly 
controlled by review requests.
Regular developers only get to use the commands that are approved by the Solr 
community leaders.

Would that work?
Thanks,
-Uv

On Jan 26 2023, at 8:09 am, Mike Drob  wrote:
> Having massive infrastructure to run PRs is pretty cool.
>
> I'm worried about letting arbitrary people run code on these
> machines though - a single 'crave run -- mine_bitcoin.exe' would ruin the
> system for everybody, or it's not hard to imagine a slightly more indirect
> case where an attacker adds a test that launches an undesirable process and
> runs crave. What safeguards exist to protect against this? At least with
> GHA we have to approve non-committers tests to run, but opening it up to
> local command line access sounds very broad.
>
> Mike
> On Wed, Jan 25, 2023 at 10:34 PM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
>
> > This is very cool. Thanks for working on this, David. Can multiple
> > developers execute their tests at the same time?
> >
> > On Thu, 26 Jan, 2023, 5:07 am Noble Paul,  wrote:
> >
> > > This is interesting.
> > >
> > > So, if the PR is merged , we will have the full test running on crave.io
> > > for every PR raised?
> > >
> > > On Thu, Jan 26, 2023 at 9:22 AM David Smiley  wrote:
> > >
> > > > We haven't been running all our tests in GitHub Actions (i.e. PR
> > > > validation) because it was too time consuming to do so. I don't recall
> > > how
> > > > slow it was when someone last tried; it's probably better now but still
> > > > slow. To make up for this, there is a GHA only for SolrJ if a PR
> > touches
> > > > SolrJ.
> > > >
> > > > There's now a PR here to introduce a new GHA that builds on Crave.io
> > on a
> > > > beefy machine: https://github.com/apache/solr/pull/1303 The PR
> > > validation
> > > > took 11 minutes which is similar to the amount of time it took a GHA to
> > > > just do precommit checks -- 10 minutes :-)
> > > > I think we can remove the SolrJ specific GHA as it'll be redundant.
> > > >
> > > > Furthermore, anyone can use this to run tests from the convenience of
> > > your
> > > > laptop at the CLI while you are in the middle of any change (doesn't
> > > matter
> > > > what you have committed or not, pushed or not). To do so, run: crave
> > run
> > > > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > > > -Ptests.jvms=48 test'
> > > >
> > > > Yeah that's long. There is a discussion in JIRA underway that may lead
> > > to
> > > > eliminating the "localSettings" step if, for example, it's moved to a
> > > bash
> > > > script executed by the gradle wrapper (my proposal). I should also be
> > > able
> > > > to configure crave with a default run configuration with this baked in.
> > > > I'll post an update when I'm able to do that.
> > > >
> > > > ~ David Smiley
> > > > Apache Lucene/Solr Search Developer
> > > > http://www.linkedin.com/in/davidwsmiley
> > > >
> > >
> > >
> > > --
> > > -
> > > Noble Paul
> > >
> >
>

Re: Running all tests via Crave.io

[Mike Drob]

Having massive infrastructure to run PRs is pretty cool.

I'm worried about letting arbitrary people run code on these
machines though - a single 'crave run -- mine_bitcoin.exe' would ruin the
system for everybody, or it's not hard to imagine a slightly more indirect
case where an attacker adds a test that launches an undesirable process and
runs crave. What safeguards exist to protect against this? At least with
GHA we have to approve non-committers tests to run, but opening it up to
local command line access sounds very broad.

Mike

On Wed, Jan 25, 2023 at 10:34 PM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> This is very cool. Thanks for working on this, David. Can multiple
> developers execute their tests at the same time?
>
> On Thu, 26 Jan, 2023, 5:07 am Noble Paul,  wrote:
>
> > This is interesting.
> >
> > So, if the PR is merged , we will have the full test running on crave.io
> > for every PR raised?
> >
> > On Thu, Jan 26, 2023 at 9:22 AM David Smiley  wrote:
> >
> > > We haven't been running all our tests in GitHub Actions (i.e. PR
> > > validation) because it was too time consuming to do so.  I don't recall
> > how
> > > slow it was when someone last tried; it's probably better now but still
> > > slow.  To make up for this, there is a GHA only for SolrJ if a PR
> touches
> > > SolrJ.
> > >
> > > There's now a PR here to introduce a new GHA that builds on Crave.io
> on a
> > > beefy machine: https://github.com/apache/solr/pull/1303  The PR
> > validation
> > > took 11 minutes which is similar to the amount of time it took a GHA to
> > > just do precommit checks -- 10 minutes :-)
> > > I think we can remove the SolrJ specific GHA as it'll be redundant.
> > >
> > > Furthermore, anyone can use this to run tests from the convenience of
> > your
> > > laptop at the CLI while you are in the middle of any change (doesn't
> > matter
> > > what you have committed or not, pushed or not).  To do so, run: crave
> run
> > > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > > -Ptests.jvms=48 test'
> > >
> > > Yeah that's long.  There is a discussion in JIRA underway that may lead
> > to
> > > eliminating the "localSettings" step if, for example, it's moved to a
> > bash
> > > script executed by the gradle wrapper (my proposal).  I should also be
> > able
> > > to configure crave with a default run configuration with this baked in.
> > > I'll post an update when I'm able to do that.
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> > >
> >
> >
> > --
> > -
> > Noble Paul
> >
>

Re: Running all tests via Crave.io

[Ishan Chattopadhyaya]

This is very cool. Thanks for working on this, David. Can multiple
developers execute their tests at the same time?

On Thu, 26 Jan, 2023, 5:07 am Noble Paul,  wrote:

> This is interesting.
>
> So, if the PR is merged , we will have the full test running on crave.io
> for every PR raised?
>
> On Thu, Jan 26, 2023 at 9:22 AM David Smiley  wrote:
>
> > We haven't been running all our tests in GitHub Actions (i.e. PR
> > validation) because it was too time consuming to do so.  I don't recall
> how
> > slow it was when someone last tried; it's probably better now but still
> > slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
> > SolrJ.
> >
> > There's now a PR here to introduce a new GHA that builds on Crave.io on a
> > beefy machine: https://github.com/apache/solr/pull/1303  The PR
> validation
> > took 11 minutes which is similar to the amount of time it took a GHA to
> > just do precommit checks -- 10 minutes :-)
> > I think we can remove the SolrJ specific GHA as it'll be redundant.
> >
> > Furthermore, anyone can use this to run tests from the convenience of
> your
> > laptop at the CLI while you are in the middle of any change (doesn't
> matter
> > what you have committed or not, pushed or not).  To do so, run: crave run
> > -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> > -Ptests.jvms=48 test'
> >
> > Yeah that's long.  There is a discussion in JIRA underway that may lead
> to
> > eliminating the "localSettings" step if, for example, it's moved to a
> bash
> > script executed by the gradle wrapper (my proposal).  I should also be
> able
> > to configure crave with a default run configuration with this baked in.
> > I'll post an update when I'm able to do that.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
>
>
> --
> -
> Noble Paul
>

Re: Running all tests via Crave.io

[Noble Paul]

This is interesting.

So, if the PR is merged , we will have the full test running on crave.io
for every PR raised?

On Thu, Jan 26, 2023 at 9:22 AM David Smiley  wrote:

> We haven't been running all our tests in GitHub Actions (i.e. PR
> validation) because it was too time consuming to do so.  I don't recall how
> slow it was when someone last tried; it's probably better now but still
> slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
> SolrJ.
>
> There's now a PR here to introduce a new GHA that builds on Crave.io on a
> beefy machine: https://github.com/apache/solr/pull/1303  The PR validation
> took 11 minutes which is similar to the amount of time it took a GHA to
> just do precommit checks -- 10 minutes :-)
> I think we can remove the SolrJ specific GHA as it'll be redundant.
>
> Furthermore, anyone can use this to run tests from the convenience of your
> laptop at the CLI while you are in the middle of any change (doesn't matter
> what you have committed or not, pushed or not).  To do so, run: crave run
> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> -Ptests.jvms=48 test'
>
> Yeah that's long.  There is a discussion in JIRA underway that may lead to
> eliminating the "localSettings" step if, for example, it's moved to a bash
> script executed by the gradle wrapper (my proposal).  I should also be able
> to configure crave with a default run configuration with this baked in.
> I'll post an update when I'm able to do that.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>


-- 
-
Noble Paul

Re: Running all tests via Crave.io

[Gus Heck]

This looks awesome. Is Crave really prepared for anyone in the world to run
this build on their hardware whenever they want? or does it require
membership/paid? If there is a limit is it per person or per project?

I don't mind a command of that length so much. That's what bash history is
for (or run configurations in intellij, etc).

On Wed, Jan 25, 2023 at 5:22 PM David Smiley  wrote:

> We haven't been running all our tests in GitHub Actions (i.e. PR
> validation) because it was too time consuming to do so.  I don't recall how
> slow it was when someone last tried; it's probably better now but still
> slow.  To make up for this, there is a GHA only for SolrJ if a PR touches
> SolrJ.
>
> There's now a PR here to introduce a new GHA that builds on Crave.io on a
> beefy machine: https://github.com/apache/solr/pull/1303  The PR validation
> took 11 minutes which is similar to the amount of time it took a GHA to
> just do precommit checks -- 10 minutes :-)
> I think we can remove the SolrJ specific GHA as it'll be redundant.
>
> Furthermore, anyone can use this to run tests from the convenience of your
> laptop at the CLI while you are in the middle of any change (doesn't matter
> what you have committed or not, pushed or not).  To do so, run: crave run
> -- './gradlew localSettings && ./gradlew --max-workers=`nproc`
> -Ptests.jvms=48 test'
>
> Yeah that's long.  There is a discussion in JIRA underway that may lead to
> eliminating the "localSettings" step if, for example, it's moved to a bash
> script executed by the gradle wrapper (my proposal).  I should also be able
> to configure crave with a default run configuration with this baked in.
> I'll post an update when I'm able to do that.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)