[jira] [Created] (HADOOP-17457) Seeing test ITestS3AInconsistency.testGetFileStatus failure.
Mukund Thakur created HADOOP-17457:
--
Summary: Seeing test ITestS3AInconsistency.testGetFileStatus
failure.
Key: HADOOP-17457
URL: https://issues.apache.org/jira/browse/HADOOP-17457
Project: Hadoop Common
Issue Type: Bug
Reporter: Mukund Thakur
{quote}[*ERROR*] *Tests* *run: 3*, *Failures: 1*, Errors: 0, Skipped: 0, Time
elapsed: 30.944 s *<<< FAILURE!* - in
org.apache.hadoop.fs.s3a.*ITestS3AInconsistency*
[*ERROR*] testGetFileStatus(org.apache.hadoop.fs.s3a.ITestS3AInconsistency)
Time elapsed: 6.471 s <<< FAILURE!
java.lang.AssertionError: getFileStatus should fail due to delayed visibility.
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.fs.s3a.ITestS3AInconsistency.testGetFileStatus(ITestS3AInconsistency.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17430) Restore ability to set Text to empty byte array
[
https://issues.apache.org/jira/browse/HADOOP-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17430.
-
Fix Version/s: 3.3.1
Resolution: Fixed
> Restore ability to set Text to empty byte array
>
>
> Key: HADOOP-17430
> URL: https://issues.apache.org/jira/browse/HADOOP-17430
> Project: Hadoop Common
> Issue Type: Wish
> Components: common
>Reporter: gaozhan ding
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> In org.apache.hadoop.io.Text:clear() method, the comments show that we can
> free the bytes by call set(new byte[0]), but it's not going to work now.
> Maybe we can follow this comments.
>
>
> {code:java}
> // org.apache.hadoop.io.Text
> /**
> * Clear the string to empty.
> *
> * Note: For performance reasons, this call does not clear the
> * underlying byte array that is retrievable via {@link #getBytes()}.
> * In order to free the byte-array memory, call {@link #set(byte[])}
> * with an empty byte array (For example, new byte[0]).
> */
> public void clear() {
> length = 0;
> textLength = -1;
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17456) S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure
Steve Loughran created HADOOP-17456:
---
Summary: S3A
ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure
Key: HADOOP-17456
URL: https://issues.apache.org/jira/browse/HADOOP-17456
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Failure in {{ITestPartialRenamesDeletes.testPartialDirDelete}}; wrong #of
delete requests.
build options: -Dparallel-tests -DtestsThreadCount=6 -Dscale -Dmarkers=delete
-Ds3guard -Ddynamo
The assert fails on a line changes in HADOOP-17271; assumption being, there are
some test run states where things happen differently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker
[
https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-17403.
-
Fix Version/s: 3.3.1
Assignee: Steve Loughran
Resolution: Workaround
> S3A ITestPartialRenamesDeletes failure: missing directory marker
>
>
> Key: HADOOP-17403
> URL: https://issues.apache.org/jira/browse/HADOOP-17403
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.3.1
>
>
> Seemingly transient failure of the test ITestPartialRenamesDeletes with the
> latest HADOOP-17244 changes in: an expected directory marker was not found.
> Test run was (unintentionally) sequential, markers=delete, s3guard on
> {code}
> -Dmarkers=delete -Ds3guard -Ddynamo -Dscale
> {code}
> Hasn't come back since.
> The bucket's retention policy was authoritative, but no dirs were declared as
> such
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16995) ITestS3AConfiguration proxy tests fail when bucket probes == 0
[
https://issues.apache.org/jira/browse/HADOOP-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-16995.
-
Resolution: Fixed
got +1 from Steve on the PR, committed to trunk
> ITestS3AConfiguration proxy tests fail when bucket probes == 0
> --
>
> Key: HADOOP-16995
> URL: https://issues.apache.org/jira/browse/HADOOP-16995
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> when bucket probes are disabled, proxy config tests in ITestS3AConfiguration
> fail because the probes aren't being attempted in initialize()
> {code}
>
> fs.s3a.bucket.probe
> 0
>
> {code}
> Cause: HADOOP-16711
> Fix: call unsetBaseAndBucketOverrides for bucket probe in test conf, then set
> the probe value to 2 just to be resilient to future default changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-17454. - Resolution: Fixed got +1 from Steve on PR, committed to trunk > [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 > --- > > Key: HADOOP-17454 > URL: https://issues.apache.org/jira/browse/HADOOP-17454 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Set the value of fs.s3a.bucket.probe to 0 by default. > Bucket existence checks are done in the initialization phase of the > S3AFileSystem. It's not required to run this check: the operation itself will > fail if the bucket does not exist instead of the check. > Some points on why do we want to set this to 0: > * When it's set to 0, bucket existence checks won't be done during > initialization thus making it faster. > * Avoid the additional one or two requests on the bucket root, so the user > does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir
Gabor Bota created HADOOP-17455:
---
Summary: [s3a] Intermittent failure of
ITestS3ADeleteCost.testDeleteSingleFileInDir
Key: HADOOP-17455
URL: https://issues.apache.org/jira/browse/HADOOP-17455
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.3.0
Reporter: Gabor Bota
Assignee: Steve Loughran
Test failed against ireland intermittently with the following config:
{{mvn clean verify -Dparallel-tests -DtestsThreadCount=8}}
xml based config in auth-keys.xml:
{code:xml}
fs.s3a.metadatastore.impl
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
Gabor Bota created HADOOP-17454: --- Summary: [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 Key: HADOOP-17454 URL: https://issues.apache.org/jira/browse/HADOOP-17454 Project: Hadoop Common Issue Type: Improvement Reporter: Gabor Bota Assignee: Gabor Bota Set the value of fs.s3a.bucket.probe to 0 in the code-default.xml. Bucket existence checks are done in the initialization phase of the S3AFileSystem. It's not required to run this check: the operation itself will fail if the bucket does not exist instead of the check. Some points on why do we want to set this to 0: * When it's set to 0, bucket existence checks won't be done during initialization thus making it faster. * Avoid the additional one or two requests on the bucket root, so the user does not need rights to read or list that folder. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
