[jira] [Resolved] (HADOOP-17312) S3AInputStream to be resilient to faiures in abort(); translate AWS Exceptions

[Yongjun Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-17312.

Resolution: Duplicate

> S3AInputStream to be resilient to faiures in abort(); translate AWS Exceptions
> --
>
> Key: HADOOP-17312
> URL: https://issues.apache.org/jira/browse/HADOOP-17312
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.0, 3.2.1
>Reporter: Steve Loughran
>Priority: Major
>
> Stack overflow issue complaining about ConnectionClosedException during 
> S3AInputStream close(), seems triggered by an EOF exception in abort. That 
> is: we are trying to close the stream and it is failing because the stream is 
> closed. oops.
> https://stackoverflow.com/questions/64412010/pyspark-org-apache-http-connectionclosedexception-premature-end-of-content-leng
> Looking @ the stack, we aren't translating AWS exceptions in abort() to IOEs, 
> which may be a factor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc

[Yongjun Zhang (Jira)]

Yongjun Zhang created HADOOP-17338:
--

 Summary: Intermittent S3AInputStream failures: Premature end of 
Content-Length delimited message body etc
 Key: HADOOP-17338
 URL: https://issues.apache.org/jira/browse/HADOOP-17338
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.3.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang


We are seeing the following exceptions intermittently when using S3AInputSteam 
(see Symptoms at the bottom).

Inspired by
https://stackoverflow.com/questions/9952815/s3-java-client-fails-a-lot-with-premature-end-of-content-length-delimited-messa
and
https://forums.aws.amazon.com/thread.jspa?threadID=83326,  we got a solution 
that has helped us, would like to put the fix to the community version.

The problem is that S3AInputStream had a short-lived S3Object which is used to 
create the wrappedSteam, and this object got garbage collected and random time, 
which caused the stream to be closed, thus the symptoms reported. 

https://github.com/aws/aws-sdk-java/blob/1.11.295/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/S3Object.java#L225
 is the s3 code that closes the stream when S3 object is garbage collected:

Here is the code in S3AInputStream that creates temporary S3Object and uses it 
to create the wrappedStream:

{code}
   S3Object object = Invoker.once(text, uri,
() -> client.getObject(request));

changeTracker.processResponse(object, operation,
targetPos);
wrappedStream = object.getObjectContent();
{code}

Symptoms:

1.
{code}
Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: 
Premature end of Content-Length delimited message body (expected: 156463674; 
received: 150001089
at 
com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
at 
com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at 
com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at 
com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at 
org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779)
at 
org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511)
at 
org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130)
at 
org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214)
at 
org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 15 more
{code}

2.
{code}
Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at 
com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at 
com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198)
at 
com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
at 
com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.j

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Yongjun Zhang]

Hi Konstantin,

Thanks for addressing my other question about failover.

Some thought to share about the suggestion Daryn made.  Seems we could try
this: let ObserverNode throws an RetriableException back to client saying
it has not reached the transaction ID to serve the client yet, maybe even
include the transaction ID gap information in the exception, then when the
client received the RetriableException, it can decide whether the continue
to send the request to the observer node again, or to the active NN when
the gap is too big.

Though saving another RPC would help the performance with the current
implementation, I expect the above mentioned exception only happens
infrequently, so the performance won't be too bad, plus the client has a
chance to try ANN when knowing that the observer is too behind at extreme
case.

I wonder how different the performance is between these two approaches in
cluster with real workload.

Comments?

--Yongjun

On Fri, Dec 7, 2018 at 4:10 PM Konstantin Shvachko 
wrote:

> Hi Daryn,
>
> Wanted to backup Chen's earlier response to your concerns about rotating
> calls in the call queue.
> Our design
> 1. targets directly the livelock problem by rejecting calls on the Observer
> that are not likely to be responded in timely matter: HDFS-13873.
> 2. The call queue rotation is only done on Observers, and never on the
> active NN, so it stays free of attacks like you suggest.
>
> If this is a satisfactory mitigation for the problem could you please
> reconsider your -1, so that people could continue voting on this thread.
>
> Thanks,
> --Konst
>
> On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:
>
> > -1 pending additional info.  After a cursory scan, I have serious
> concerns
> > regarding the design.  This seems like a feature that should have been
> > purely implemented in hdfs w/o touching the common IPC layer.
> >
> > The biggest issue in the alignment context.  It's purpose appears to be
> > for allowing handlers to reinsert calls back into the call queue.  That's
> > completely unacceptable.  A buggy or malicious client can easily cause
> > livelock in the IPC layer with handlers only looping on calls that never
> > satisfy the condition.  Why is this not implemented via
> RetriableExceptions?
> >
> > On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang  >
> > wrote:
> >
> >> Great work guys.
> >>
> >> Wonder if we can elaborate what's impact of not having #2 fixed, and why
> >> #2
> >> is not needed for the feature to complete?
> >> 2. Need to fix automatic failover with ZKFC. Currently it does not
> doesn't
> >> know about ObserverNodes trying to convert them to SBNs.
> >>
> >> Thanks.
> >> --Yongjun
> >>
> >>
> >> On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko <
> shv.had...@gmail.com>
> >> wrote:
> >>
> >> > Hi Hadoop developers,
> >> >
> >> > I would like to propose to merge to trunk the feature branch
> HDFS-12943
> >> for
> >> > Consistent Reads from Standby Node. The feature is intended to scale
> >> read
> >> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> >> > NameNode. We should be able to accommodate higher overall RPC
> workloads
> >> (up
> >> > to 4x by some estimates) by adding multiple ObserverNodes.
> >> >
> >> > The main functionality has been implemented see sub-tasks of
> HDFS-12943.
> >> > We followed up with the test plan. Testing was done on two independent
> >> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> >> > We ran standard HDFS commands, MR jobs, admin commands including
> manual
> >> > failover.
> >> > We know of one cluster running this feature in production.
> >> >
> >> > There are a few outstanding issues:
> >> > 1. Need to provide proper documentation - a user guide for the new
> >> feature
> >> > 2. Need to fix automatic failover with ZKFC. Currently it does not
> >> doesn't
> >> > know about ObserverNodes trying to convert them to SBNs.
> >> > 3. Scale testing and performance fine-tuning
> >> > 4. As testing progresses, we continue fixing non-critical bugs like
> >> > HDFS-14116.
> >> >
> >> > I attached a unified patch to the umbrella jira for the review and
> >> Jenkins
> >> > build.
> >> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> >> 12.
> >> >
> >> > Thanks,
> >> > --Konstantin
> >> >
> >>
> >
> >
> > --
> >
> > Daryn
> >
>

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Yongjun Zhang]

Great work guys.

Wonder if we can elaborate what's impact of not having #2 fixed, and why #2
is not needed for the feature to complete?
2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
know about ObserverNodes trying to convert them to SBNs.

Thanks.
--Yongjun


On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
>
> I attached a unified patch to the umbrella jira for the review and Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 12.
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Hi Vinod,

Thanks a lot for the finding and fixing! Sorry for having missed your email
earlier.

Best,

--Yongjun

On Mon, Aug 13, 2018 at 9:51 AM, Vinod Kumar Vavilapalli  wrote:

> Yongjun,
>
> Looks like you didn't add the links to 3.0.3 binary release on the
> http://hadoop.apache.org/releases.html page.
>
> I just did it, FYI: https://svn.apache.org/viewvc?view=revision=
> 1837967
>
> Thanks
> +Vinod
>
>
> On May 31, 2018, at 10:48 PM, Yongjun Zhang  wrote:
>
> Greetings all,
>
> I've created the first release candidate (RC0) for Apache Hadoop
> 3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
> about 249
> important fixes and improvements, among which there are 8 blockers. See
> https://issues.apache.org/jira/issues/?filter=12343997
>
> The RC artifacts are available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>
> The maven artifacts are available via
> https://repository.apache.org/content/repositories/orgapachehadoop-1126
>
> Please try the release and vote; the vote will run for the usual 5 working
> days, ending on 06/07/2018 PST time. Would really appreciate your
> participation here.
>
> I bumped into quite some issues along the way, many thanks to quite a few
> people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.
>
> Thanks,
>
> --Yongjun
>
>
>

[jira] [Created] (HADOOP-15720) rpcTimeout may not have been applied correctly

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-15720:
--

 Summary: rpcTimeout may not have been applied correctly
 Key: HADOOP-15720
 URL: https://issues.apache.org/jira/browse/HADOOP-15720
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Reporter: Yongjun Zhang


org.apache.hadoop.ipc.Client send multiple RPC calls to server synchronously 
via the same connection as in the following synchronized code block:
{code:java}
  synchronized (sendRpcRequestLock) {
Future senderFuture = sendParamsExecutor.submit(new Runnable() {
  @Override
  public void run() {
try {
  synchronized (Connection.this.out) {
if (shouldCloseConnection.get()) {
  return;
}

if (LOG.isDebugEnabled()) {
  LOG.debug(getName() + " sending #" + call.id
  + " " + call.rpcRequest);
}
 
byte[] data = d.getData();
int totalLength = d.getLength();
out.writeInt(totalLength); // Total Length
out.write(data, 0, totalLength);// RpcRequestHeader + RpcRequest
out.flush();
  }
} catch (IOException e) {
  // exception at this point would leave the connection in an
  // unrecoverable state (eg half a call left on the wire).
  // So, close the connection, killing any outstanding calls
  markClosed(e);
} finally {
  //the buffer is just an in-memory buffer, but it is still polite 
to
  // close early
  IOUtils.closeStream(d);
}
  }
});
  
try {
  senderFuture.get();
} catch (ExecutionException e) {
  Throwable cause = e.getCause();
  
  // cause should only be a RuntimeException as the Runnable above
  // catches IOException
  if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
  } else {
throw new RuntimeException("unexpected checked exception", cause);
  }
}
  }
{code}
And it then waits for result asynchronously via
{code:java}
/* Receive a response.
 * Because only one receiver, so no synchronization on in.
 */
private void receiveRpcResponse() {
  if (shouldCloseConnection.get()) {
return;
  }
  touch();
  
  try {
int totalLen = in.readInt();
RpcResponseHeaderProto header = 
RpcResponseHeaderProto.parseDelimitedFrom(in);
checkResponse(header);

int headerLen = header.getSerializedSize();
headerLen += CodedOutputStream.computeRawVarint32Size(headerLen);

int callId = header.getCallId();
if (LOG.isDebugEnabled())
  LOG.debug(getName() + " got value #" + callId);

Call call = calls.get(callId);
RpcStatusProto status = header.getStatus();
..
{code}
However, we can see that the {{call}} returned by {{receiveRpcResonse()}} above 
may be in any order.

The following code
{code:java}
int totalLen = in.readInt();
{code}
eventually calls one of the following two methods, where rpcTimeOut is checked 
against:
{code:java}
  /** Read a byte from the stream.
   * Send a ping if timeout on read. Retries if no failure is detected
   * until a byte is read.
   * @throws IOException for any IO problem other than socket timeout
   */
  @Override
  public int read() throws IOException {
int waiting = 0;
do {
  try {
return super.read();
  } catch (SocketTimeoutException e) {
waiting += soTimeout;
handleTimeout(e, waiting);
  }
} while (true);
  }

  /** Read bytes into a buffer starting from offset off
   * Send a ping if timeout on read. Retries if no failure is detected
   * until a byte is read.
   * 
   * @return the total number of bytes read; -1 if the connection is closed.
   */
  @Override
  public int read(byte[] buf, int off, int len) throws IOException {
int waiting = 0;
do {
  try {
return super.read(buf, off, len);
  } catch (SocketTimeoutException e) {
waiting += soTimeout;
handleTimeout(e, waiting);
  }
} while (true);
  }
{code}
But the waiting time is always initialized to 0 for each of the above read 
calls, so each call can take up to rpcTimeout. And the real time to time out a 
call appears to be accumulative.

For example, if the client issue call1, call2, then it waits for result, if the 
first call1 took (rpcTimeout - 1), thus no time out, the secon

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Welcome Jonathan.

http://hadoop.apache.org/releases.html stated:
"Hadoop is released as source code tarballs with corresponding binary
tarballs for convenience. "

and Andrew Wang said "The binary artifacts (including JARs) are technically
just convenience artifacts" and it seems not an uncommon practice to do
follow-up builds to release maven artifacts.

IIRC, Andrew once shared with me that we started in 3.x to use a single
build to to do both release binaries creation and maven artifacts
deployment, prior releases are using multiple builds:

Referring to https://wiki.apache.org/hadoop/HowToRelease

   - 3.x: step 4 in   "Creating the release candidate (X.Y.Z-RC)"
   section does both release binaries creation and maven artifacts deployment.
   - prior to 3.x: step 4 does release binary creation, and step 10 does
   maven artifacts deployment, *each step does its build so two builds here*.
   As a matter of fact, I did not run step 10 for 3.0.3.

That said, I agree that ideally it's better to do a single build to
generate release binaries and deploy maven artifacts from the same build.

Hope it helps. Welcome other folks to chime in.

Best,

--Yongjun






On Mon, Jul 9, 2018 at 2:08 PM, Jonathan Eagles  wrote:

> Thank you, Yongjun Zhang for resolving this issue for me. I have verified
> the 3.0.3 build is now working for me for tez to specify as a hadoop
> dependency.
>
> As for release procedure, can someone comment on what to do now that the
> artifacts published to maven are different than the voted on artifacts. I
> believe the source code is what is voted on and the maven artifacts are
> just for convenience, but would like an "official" answer.
>
> Reference:
> https://issues.apache.org/jira/browse/TEZ-3955
>
> Regards,
> jeagles
>
> On Mon, Jul 9, 2018 at 12:26 PM, Yongjun Zhang 
> wrote:
>
>> HI Jonathan,
>>
>> I have updated the artifacts, so now
>>
>> https://repository.apache.org/#nexus-search;gav~org.apache.hadoop~~3
>> .0.2~~
>> https://repository.apache.org/#nexus-search;gav~org.apache.hadoop~~3.0.3
>> ~~
>>
>> are more consistent, except that 3.0.3 has an extra entry for rbf. Would
>> you please try again?
>>
>> The propagation to
>> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project
>> will take some time. I did nothing different than last time, so keep
>> finger crossed that it will propagate there.
>>
>> Thanks Sammi Chen and Andrew Wang for info and advice, and sorry for the
>> inconvenience again.
>>
>> Best,
>>
>> --Yongjun
>>
>> On Mon, Jul 2, 2018 at 9:30 AM, Jonathan Eagles 
>> wrote:
>>
>>> Release 3.0.3 is still broken due to the missing artifacts. Any update
>>> on when these artifacts will be published?
>>>
>>> On Wed, Jun 27, 2018 at 8:25 PM, Chen, Sammi 
>>> wrote:
>>>
>>>> Hi Yongjun,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The artifacts will be pushed to https://mvnrepository.com/arti
>>>> fact/org.apache.hadoop/hadoop-project after step 6 of Publishing steps.
>>>>
>>>>
>>>> For 2.9.1, I remember I absolutely did the step before. I redo the step
>>>> 6 today and now 2.9.1 is pushed to the mvn repo.
>>>>
>>>> You can double check it. I suspect sometimes Nexus may fail to notify
>>>> user when this is unexpected failures.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Bests,
>>>>
>>>> Sammi
>>>>
>>>> *From:* Yongjun Zhang [mailto:yzh...@cloudera.com]
>>>> *Sent:* Sunday, June 17, 2018 12:17 PM
>>>> *To:* Jonathan Eagles ; Chen, Sammi <
>>>> sammi.c...@intel.com>
>>>> *Cc:* Eric Payne ; Hadoop Common <
>>>> common-dev@hadoop.apache.org>; Hdfs-dev ;
>>>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>>>> *Subject:* Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
>>>>
>>>>
>>>>
>>>> + Junping, Sammi
>>>>
>>>>
>>>>
>>>> Hi Jonathan,
>>>>
>>>>
>>>>
>>>> Many thanks for reporting the issues and sorry for the inconvenience.
>>>>
>>>>
>>>>
>>>> 1. Shouldn't the build be looking for artifacts in
>>>>
>>>>
>>>>
>>>> https://repository.apache.org/content/repositories/releases
>>>>
>>>> rather than
>>>>
>>&g

[jira] [Created] (HADOOP-15590) Two gpg related errors when doing hadoop release

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-15590:
--

 Summary: Two gpg related errors when doing hadoop release
 Key: HADOOP-15590
 URL: https://issues.apache.org/jira/browse/HADOOP-15590
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang


When doing 3.0.3 release, by running command

dev-support/bin/create-release --asfrelease --docker --dockercache

documented in

https://wiki.apache.org/hadoop/HowToRelease

I hit the following problems:

1. 
{quote}
starting gpg agent ERROR: Unable to launch or acquire gpg-agent. Disable 
signing.
{quote}
The script expect GPG_AGENT_INFO env being set with needed info by the 
gpg-agent. However, it was not. This is because of changes made in gpg-agent. I 
found the workaround is to add the following line to 
dev-support/bin/create-release script right after starting gpg-agent:
{quote}
export GPG_AGENT_INFO="~/.gnupg/S.gpg-agent:$(pgrep gpg-agent):1"
{quote}

2.
{quote}
gpg: can't connect to `~/.gnupg/S.gpg-agent': invalid value
{quote}
I found that this is caused by unmatching gpg-agent and gpg versions installed 
via Docker. I modified dev-support/docker/Dockerfile to install gnupg2 instead 
of gnupg. This made gpg and gpg-agent both 2.1.11 instead of one on 2.1.11 the 
other on 1.14. And this solved the above problem. 







--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

HI Jonathan,

I have updated the artifacts, so now

https://repository.apache.org/#nexus-search;gav~org.apache.hadoop~~3.0.2~~
https://repository.apache.org/#nexus-search;gav~org.apache.hadoop~~3.0.3~~

are more consistent, except that 3.0.3 has an extra entry for rbf. Would
you please try again?

The propagation to
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project
will take some time. I did nothing different than last time, so keep finger
crossed that it will propagate there.

Thanks Sammi Chen and Andrew Wang for info and advice, and sorry for the
inconvenience again.

Best,

--Yongjun

On Mon, Jul 2, 2018 at 9:30 AM, Jonathan Eagles  wrote:

> Release 3.0.3 is still broken due to the missing artifacts. Any update on
> when these artifacts will be published?
>
> On Wed, Jun 27, 2018 at 8:25 PM, Chen, Sammi  wrote:
>
>> Hi Yongjun,
>>
>>
>>
>>
>>
>> The artifacts will be pushed to https://mvnrepository.com/arti
>> fact/org.apache.hadoop/hadoop-project after step 6 of Publishing steps.
>>
>>
>> For 2.9.1, I remember I absolutely did the step before. I redo the step 6
>> today and now 2.9.1 is pushed to the mvn repo.
>>
>> You can double check it. I suspect sometimes Nexus may fail to notify
>> user when this is unexpected failures.
>>
>>
>>
>>
>>
>> Bests,
>>
>> Sammi
>>
>> *From:* Yongjun Zhang [mailto:yzh...@cloudera.com]
>> *Sent:* Sunday, June 17, 2018 12:17 PM
>> *To:* Jonathan Eagles ; Chen, Sammi <
>> sammi.c...@intel.com>
>> *Cc:* Eric Payne ; Hadoop Common <
>> common-dev@hadoop.apache.org>; Hdfs-dev ;
>> mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
>> *Subject:* Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
>>
>>
>>
>> + Junping, Sammi
>>
>>
>>
>> Hi Jonathan,
>>
>>
>>
>> Many thanks for reporting the issues and sorry for the inconvenience.
>>
>>
>>
>> 1. Shouldn't the build be looking for artifacts in
>>
>>
>>
>> https://repository.apache.org/content/repositories/releases
>>
>> rather than
>>
>>
>>
>> https://repository.apache.org/content/repositories/snapshots
>>
>> ?
>>
>>
>>
>> 2.
>>
>> Not seeing the artifact published here as well.
>>
>> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project
>>
>>
>>
>> Indeed, I did not see 2.9.1 there too. So included Sammi Chen.
>>
>>
>>
>> Hi Junping, would you please share which step in
>>
>> https://wiki.apache.org/hadoop/HowToRelease
>>
>> should have done this?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> --Yongjun
>>
>>
>>
>> On Fri, Jun 15, 2018 at 10:52 PM, Jonathan Eagles 
>> wrote:
>>
>> Upgraded Tez dependency to hadoop 3.0.3 and found this issue. Anyone else
>> seeing this issue?
>>
>>
>>
>> [ERROR] Failed to execute goal on project hadoop-shim: Could not resolve
>> dependencies for project org.apache.tez:hadoop-shim:jar:0.10.0-SNAPSHOT:
>> Failed to collect dependencies at 
>> org.apache.hadoop:hadoop-yarn-api:jar:3.0.3:
>> Failed to read artifact descriptor for 
>> org.apache.hadoop:hadoop-yarn-api:jar:3.0.3:
>> Could not find artifact org.apache.hadoop:hadoop-project:pom:3.0.3 in
>> apache.snapshots.https (https://repository.apache.org
>> /content/repositories/snapshots) -> [Help 1]
>>
>> [ERROR]
>>
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>>
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>>
>> [ERROR]
>>
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>>
>> [ERROR] [Help 1] http://cwiki.apache.org/conflu
>> ence/display/MAVEN/DependencyResolutionException
>>
>> [ERROR]
>>
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>>
>> [ERROR]   mvn  -rf :hadoop-shim
>>
>>
>>
>> Not seeing the artifact published here as well.
>>
>> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project
>>
>>
>>
>> On Tue, Jun 12, 2018 at 6:44 PM, Yongjun Zhang 
>> wrote:
>>
>> Thanks Eric!
>>
>> --Yongjun
>>
>>
>> On Mon, Jun 11, 2018 at 8:05 AM, Eric Payne 
>> wrote:
>>
>> > Sorry, Yongjun. My

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

+ Junping, Sammi

Hi Jonathan,

Many thanks for reporting the issues and sorry for the inconvenience.

1. Shouldn't the build be looking for artifacts in

https://repository.apache.org/content/repositories/releases

rather than

https://repository.apache.org/content/repositories/snapshots
?

2.

Not seeing the artifact published here as well.
https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project


Indeed, I did not see 2.9.1 there too. So included Sammi Chen.

Hi Junping, would you please share which step in
https://wiki.apache.org/hadoop/HowToRelease
should have done this?

Thanks a lot.

--Yongjun

On Fri, Jun 15, 2018 at 10:52 PM, Jonathan Eagles  wrote:

> Upgraded Tez dependency to hadoop 3.0.3 and found this issue. Anyone else
> seeing this issue?
>
> [ERROR] Failed to execute goal on project hadoop-shim: Could not resolve
> dependencies for project org.apache.tez:hadoop-shim:jar:0.10.0-SNAPSHOT:
> Failed to collect dependencies at org.apache.hadoop:hadoop-yarn-api:jar:3.0.3:
> Failed to read artifact descriptor for 
> org.apache.hadoop:hadoop-yarn-api:jar:3.0.3:
> Could not find artifact org.apache.hadoop:hadoop-project:pom:3.0.3 in
> apache.snapshots.https (https://repository.apache.
> org/content/repositories/snapshots) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/
> DependencyResolutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :hadoop-shim
>
> Not seeing the artifact published here as well.
> https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-project
>
> On Tue, Jun 12, 2018 at 6:44 PM, Yongjun Zhang 
> wrote:
>
>> Thanks Eric!
>>
>> --Yongjun
>>
>> On Mon, Jun 11, 2018 at 8:05 AM, Eric Payne 
>> wrote:
>>
>> > Sorry, Yongjun. My +1 is also binding
>> > +1 (binding)
>> > -Eric Payne
>> >
>> > On Friday, June 1, 2018, 12:25:36 PM CDT, Eric Payne <
>> > eric.payne1...@yahoo.com> wrote:
>> >
>> >
>> >
>> >
>> > Thanks a lot, Yongjun, for your hard work on this release.
>> >
>> > +1
>> > - Built from source
>> > - Installed on 6 node pseudo cluster
>> >
>> >
>> > Tested the following in the Capacity Scheduler:
>> > - Verified that running apps in labelled queues restricts tasks to the
>> > labelled nodes.
>> > - Verified that various queue config properties for CS are refreshable
>> > - Verified streaming jobs work as expected
>> > - Verified that user weights work as expected
>> > - Verified that FairOrderingPolicy in a CS queue will evenly assign
>> > resources
>> > - Verified running yarn shell application runs as expected
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Friday, June 1, 2018, 12:48:26 AM CDT, Yongjun Zhang <
>> > yjzhan...@apache.org> wrote:
>> >
>> >
>> >
>> >
>> >
>> > Greetings all,
>> >
>> > I've created the first release candidate (RC0) for Apache Hadoop
>> > 3.0.3. This is our next maintenance release to follow up 3.0.2. It
>> includes
>> > about 249
>> > important fixes and improvements, among which there are 8 blockers. See
>> > https://issues.apache.org/jira/issues/?filter=12343997
>> >
>> > The RC artifacts are available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>> >
>> > The maven artifacts are available via
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1126
>> >
>> > Please try the release and vote; the vote will run for the usual 5
>> working
>> > days, ending on 06/07/2018 PST time. Would really appreciate your
>> > participation here.
>> >
>> > I bumped into quite some issues along the way, many thanks to quite a
>> few
>> > people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy
>> Xu.
>> >
>> > Thanks,
>> >
>> > --Yongjun
>> >
>>
>
>

[jira] [Created] (HADOOP-15538) Possible dead lock in Client

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-15538:
--

 Summary: Possible dead lock in Client
 Key: HADOOP-15538
 URL: https://issues.apache.org/jira/browse/HADOOP-15538
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Reporter: Yongjun Zhang


We have a jstack collection that spans 13 minutes. Once frame per ~1.5 minutes. 
And for each of the frame, I observed the following:

{code}
Found one Java-level deadlock:
=
"IPC Parameter Sending Thread #294":
  waiting to lock monitor 0x7f68f21f3188 (object 0x000621745390, a 
java.lang.Object),
  which is held by UNKNOWN_owner_addr=0x7f68332e2800

Java stack information for the threads listed above:
===
"IPC Parameter Sending Thread #294":
at 
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:268)
- waiting to lock <0x000621745390> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:461)
- locked <0x000621745380> (a java.lang.Object)
at 
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
- locked <0x000621749850> (a java.io.BufferedOutputStream)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:1072)
- locked <0x00062174b878> (a java.io.DataOutputStream)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Found one Java-level deadlock:
=
"IPC Client (297602875) connection to x.y.z.p:8020 from impala":
  waiting to lock monitor 0x7f68f21f3188 (object 0x000621745390, a 
java.lang.Object),
  which is held by UNKNOWN_owner_addr=0x7f68332e2800

Java stack information for the threads listed above:
===
"IPC Client (297602875) connection to x.y.z.p:8020 from impala":
at 
sun.nio.ch.SocketChannelImpl.readerCleanup(SocketChannelImpl.java:279)
- waiting to lock <0x000621745390> (a java.lang.Object)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:390)
- locked <0x000621745370> (a java.lang.Object)
at 
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at 
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:553)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x0006217476f0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at 
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1113)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1006)

Found 2 deadlocks.
{code}

This happens with jdk1.8.0_162, and the code appears to match 
https://insight.io/github.com/AdoptOpenJDK/openjdk-jdk8u/tree/dev/jdk/src/share/classes/sun/nio/ch/SocketChannelImpl.java.

The first thread is blocked at:

https://insight.io/github.com/AdoptOpenJDK/openjdk-jdk8u/blob/dev/jdk/src/share/classes/sun/nio/ch/SocketChannelImpl.java?line=268

The second thread is blocked at:
https://insight.io/github.com/AdoptOpenJDK/openjdk-jdk8u/blob/dev/jdk/src/share/classes/sun/nio/ch/SocketChannelImpl.java?line=279

There are two issues here:

1. There seems to be a real deadlock because the stacks remain the same even if 
the first an last jstack frames captured is 13 minutes apart.

2. java deadlock report seems to be problematic, two t

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Thanks Eric!

--Yongjun

On Mon, Jun 11, 2018 at 8:05 AM, Eric Payne  wrote:

> Sorry, Yongjun. My +1 is also binding
> +1 (binding)
> -Eric Payne
>
> On Friday, June 1, 2018, 12:25:36 PM CDT, Eric Payne <
> eric.payne1...@yahoo.com> wrote:
>
>
>
>
> Thanks a lot, Yongjun, for your hard work on this release.
>
> +1
> - Built from source
> - Installed on 6 node pseudo cluster
>
>
> Tested the following in the Capacity Scheduler:
> - Verified that running apps in labelled queues restricts tasks to the
> labelled nodes.
> - Verified that various queue config properties for CS are refreshable
> - Verified streaming jobs work as expected
> - Verified that user weights work as expected
> - Verified that FairOrderingPolicy in a CS queue will evenly assign
> resources
> - Verified running yarn shell application runs as expected
>
>
>
>
>
>
>
> On Friday, June 1, 2018, 12:48:26 AM CDT, Yongjun Zhang <
> yjzhan...@apache.org> wrote:
>
>
>
>
>
> Greetings all,
>
> I've created the first release candidate (RC0) for Apache Hadoop
> 3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
> about 249
> important fixes and improvements, among which there are 8 blockers. See
> https://issues.apache.org/jira/issues/?filter=12343997
>
> The RC artifacts are available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>
> The maven artifacts are available via
> https://repository.apache.org/content/repositories/orgapachehadoop-1126
>
> Please try the release and vote; the vote will run for the usual 5 working
> days, ending on 06/07/2018 PST time. Would really appreciate your
> participation here.
>
> I bumped into quite some issues along the way, many thanks to quite a few
> people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.
>
> Thanks,
>
> --Yongjun
>

[jira] [Created] (HADOOP-15530) RPC could stuck at senderFuture.get()

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-15530:
--

 Summary: RPC could stuck at senderFuture.get()
 Key: HADOOP-15530
 URL: https://issues.apache.org/jira/browse/HADOOP-15530
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Reporter: Yongjun Zhang


In Client.java, sendRpcRequest does the following

{code}
   /** Initiates a rpc call by sending the rpc request to the remote server.
 * Note: this is not called from the Connection thread, but by other
 * threads.
 * @param call - the rpc request
 */
public void sendRpcRequest(final Call call)
throws InterruptedException, IOException {
  if (shouldCloseConnection.get()) {
return;
  }

  // Serialize the call to be sent. This is done from the actual
  // caller thread, rather than the sendParamsExecutor thread,

  // so that if the serialization throws an error, it is reported
  // properly. This also parallelizes the serialization.
  //
  // Format of a call on the wire:
  // 0) Length of rest below (1 + 2)
  // 1) RpcRequestHeader  - is serialized Delimited hence contains length
  // 2) RpcRequest
  //
  // Items '1' and '2' are prepared here. 
  RpcRequestHeaderProto header = ProtoUtil.makeRpcRequestHeader(
  call.rpcKind, OperationProto.RPC_FINAL_PACKET, call.id, call.retry,
  clientId);

  final ResponseBuffer buf = new ResponseBuffer();
  header.writeDelimitedTo(buf);
  RpcWritable.wrap(call.rpcRequest).writeTo(buf);

  synchronized (sendRpcRequestLock) {
Future senderFuture = sendParamsExecutor.submit(new Runnable() {
  @Override
  public void run() {
try {
  synchronized (ipcStreams.out) {
if (shouldCloseConnection.get()) {
  return;
}
if (LOG.isDebugEnabled()) {
  LOG.debug(getName() + " sending #" + call.id
  + " " + call.rpcRequest);
}
// RpcRequestHeader + RpcRequest
ipcStreams.sendRequest(buf.toByteArray());
ipcStreams.flush();
  }
} catch (IOException e) {
  // exception at this point would leave the connection in an
  // unrecoverable state (eg half a call left on the wire).
  // So, close the connection, killing any outstanding calls
  markClosed(e);
} finally {
  //the buffer is just an in-memory buffer, but it is still polite 
to
  // close early
  IOUtils.closeStream(buf);
}
  }
});

try {
  senderFuture.get();
} catch (ExecutionException e) {
  Throwable cause = e.getCause();

  // cause should only be a RuntimeException as the Runnable above
  // catches IOException
  if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
  } else {
throw new RuntimeException("unexpected checked exception", cause);
  }
}
  }
}
{code}

It's observed that the call can be stuck at {{senderFuture.get();}}

Given that we support rpcTimeOut, we could chose the second method of Future 
below:
{code}
  /**
 * Waits if necessary for the computation to complete, and then
 * retrieves its result.
 *
 * @return the computed result
 * @throws CancellationException if the computation was cancelled
 * @throws ExecutionException if the computation threw an
 * exception
 * @throws InterruptedException if the current thread was interrupted
 * while waiting
 */
V get() throws InterruptedException, ExecutionException;

/**
 * Waits if necessary for at most the given time for the computation
 * to complete, and then retrieves its result, if available.
 *
 * @param timeout the maximum time to wait
 * @param unit the time unit of the timeout argument
 * @return the computed result
 * @throws CancellationException if the computation was cancelled
 * @throws ExecutionException if the computation threw an
 * exception
 * @throws InterruptedException if the current thread was interrupted
 * while waiting
 * @throws TimeoutException if the wait timed out
 */
V get(long timeout, TimeUnit unit)
throws InterruptedException, ExecutionException, TimeoutException;
{code}

In theory, since the RPC at client is serialized, we could just use the main 
thread to do the execution, instead of using a threadpool to create new thread. 
This can be discussed in a separate jira.




  



--
This message was sent b

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Many thanks to all of you!

Now we have many +1s including 3 from PMCs, and no -1s. Would like to call
it a done vote!

I will work on the remaining steps to push forward the release.

Wish you all a very nice weekend,

--Yongjun




On Fri, Jun 8, 2018 at 2:45 PM, John Zhuge  wrote:

> Thanks Xongjun for the excellent work to drive this release!
>
>
> +1 (binding)
>
>- Verified checksums and signatures of tarballs
>- Built source with native, Oracle Java 1.8.0_152 on Mac OS X 10.13.5
>- Verified cloud connectors:
>   - ADLS integration tests passed with 1 failure, not a blocker
>- Deployed both binary and built source to a pseudo cluster, passed
>the following sanity tests in insecure and SSL mode:
>   - HDFS basic and ACL
>   - WebHDFS CLI ls and REST LISTSTATUS
>   - DistCp basic
>   - MapReduce wordcount
>   - KMS and HttpFS basic and servlets
>   - Balancer start/stop
>
>
> ADLS unit test failure:
>
>
> [ERROR] Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 68.889 s <<< FAILURE! - in org.apache.hadoop.fs.adl.live.
> TestAdlFileSystemContractLive
>
> [ERROR] testMkdirsWithUmask(org.apache.hadoop.fs.adl.live.
> TestAdlFileSystemContractLive)  Time elapsed: 0.851 s  <<< FAILURE!
>
> java.lang.AssertionError: expected:<461> but was:<456>
>
>
> See https://issues.apache.org/jira/browse/HADOOP-14435. I don't think it
> is a blocker.
>
> Thanks,
>
> On Fri, Jun 8, 2018 at 12:04 PM, Xiao Chen  wrote:
>
>> Thanks for the effort on this Yongjun.
>>
>> +1 (binding)
>>
>>- Built from src
>>- Deployed a pseudo distributed HDFS with KMS
>>- Ran basic hdfs commands with encryption
>>- Sanity checked webui and logs
>>
>>
>>
>> -Xiao
>>
>> On Fri, Jun 8, 2018 at 10:34 AM, Brahma Reddy Battula <
>> brahmareddy.batt...@hotmail.com> wrote:
>>
>> > Thanks yongjun zhang for driving this release.
>> >
>> > +1 (binding).
>> >
>> >
>> > ---Built from the source
>> > ---Installed HA cluster
>> > ---Execute the basic shell commands
>> > ---Browsed the UI's
>> > ---Ran sample jobs like pi,wordcount
>> >
>> >
>> >
>> > 
>> > From: Yongjun Zhang 
>> > Sent: Friday, June 8, 2018 1:04 PM
>> > To: Allen Wittenauer
>> > Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>> > yarn-...@hadoop.apache.org
>> > Subject: Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
>> >
>> > BTW, thanks Allen and Steve for discussing and suggestion about the site
>> > build problem I hit earlier, I did the following step
>> >
>> > mvn install -DskipTests
>> >
>> > before doing the steps Nanda listed helped to solve the problems.
>> >
>> > --Yongjun
>> >
>> >
>> >
>> >
>> > On Thu, Jun 7, 2018 at 6:15 PM, Yongjun Zhang 
>> wrote:
>> >
>> > > Thank you all very much for the testing, feedback and discussion!
>> > >
>> > > I was able to build outside docker, by following the steps Nanda
>> > > described, I saw the same problem; then I tried 3.0.2 released a while
>> > > back, it has the same issue.
>> > >
>> > > As Allen pointed out, it seems the steps to build site are not
>> correct. I
>> > > have not figured out the correct steps yet.
>> > >
>> > > At this point, I think this issue should not block the 3.0.3 issue.
>> While
>> > > at the same time we need to figure out the right steps to build the
>> site.
>> > > Would you please let me know if you think differently?
>> > >
>> > > We only have the site build issue reported so far. And we don't have
>> > > enough PMC votes yet. So need some more PMCs to help.
>> > >
>> > > Thanks again, and best regards,
>> > >
>> > > --Yongjun
>> > >
>> > >
>> > > On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer <
>> > a...@effectivemachines.com
>> > > > wrote:
>> > >
>> > >> > On Jun 7, 2018, at 11:47 AM, Steve Loughran <
>> ste...@hortonworks.com>
>> > >> wrote:
>> > >> >
>> > >> > Actually, Yongjun has been really good at helping me get set up
>> for a
>> > >> 2.7.7 release, including "things you need to do to get GPG working in
>> > the
>> > >> docker image”
>> > >>
>> > >> *shrugs* I use a different release script after some changes
>> > >> broke the in-tree version for building on OS X and I couldn’t get the
>> > fixes
>> > >> committed upstream.  So not sure what the problems are that you are
>> > hitting.
>> > >>
>> > >> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu <
>> > >> nvadiv...@hortonworks.com> wrote:
>> > >> >
>> > >> > It will be helpful if we can get the correct steps, and also update
>> > the
>> > >> wiki.
>> > >> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>> > >> Release+Validation
>> > >>
>> > >> Yup. Looking forward to seeing it.
>> > >> 
>> -
>> > >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> > >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> > >>
>> > >>
>> > >
>> >
>>
>
>
>
> --
> John
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Hello,

PMC votes are needed for release, can I get some help from PMCs?

Really appreciate it!

--Yongjun

On Thu, Jun 7, 2018 at 6:15 PM, Yongjun Zhang  wrote:

> Thank you all very much for the testing, feedback and discussion!
>
> I was able to build outside docker, by following the steps Nanda
> described, I saw the same problem; then I tried 3.0.2 released a while
> back, it has the same issue.
>
> As Allen pointed out, it seems the steps to build site are not correct. I
> have not figured out the correct steps yet.
>
> At this point, I think this issue should not block the 3.0.3 issue. While
> at the same time we need to figure out the right steps to build the site.
> Would you please let me know if you think differently?
>
> We only have the site build issue reported so far. And we don't have
> enough PMC votes yet. So need some more PMCs to help.
>
> Thanks again, and best regards,
>
> --Yongjun
>
>
> On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer  > wrote:
>
>> > On Jun 7, 2018, at 11:47 AM, Steve Loughran 
>> wrote:
>> >
>> > Actually, Yongjun has been really good at helping me get set up for a
>> 2.7.7 release, including "things you need to do to get GPG working in the
>> docker image”
>>
>> *shrugs* I use a different release script after some changes
>> broke the in-tree version for building on OS X and I couldn’t get the fixes
>> committed upstream.  So not sure what the problems are that you are hitting.
>>
>> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu <
>> nvadiv...@hortonworks.com> wrote:
>> >
>> > It will be helpful if we can get the correct steps, and also update the
>> wiki.
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>> Release+Validation
>>
>> Yup. Looking forward to seeing it.
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

BTW, thanks Allen and Steve for discussing and suggestion about the site
build problem I hit earlier, I did the following step

mvn install -DskipTests

before doing the steps Nanda listed helped to solve the problems.

--Yongjun




On Thu, Jun 7, 2018 at 6:15 PM, Yongjun Zhang  wrote:

> Thank you all very much for the testing, feedback and discussion!
>
> I was able to build outside docker, by following the steps Nanda
> described, I saw the same problem; then I tried 3.0.2 released a while
> back, it has the same issue.
>
> As Allen pointed out, it seems the steps to build site are not correct. I
> have not figured out the correct steps yet.
>
> At this point, I think this issue should not block the 3.0.3 issue. While
> at the same time we need to figure out the right steps to build the site.
> Would you please let me know if you think differently?
>
> We only have the site build issue reported so far. And we don't have
> enough PMC votes yet. So need some more PMCs to help.
>
> Thanks again, and best regards,
>
> --Yongjun
>
>
> On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer  > wrote:
>
>> > On Jun 7, 2018, at 11:47 AM, Steve Loughran 
>> wrote:
>> >
>> > Actually, Yongjun has been really good at helping me get set up for a
>> 2.7.7 release, including "things you need to do to get GPG working in the
>> docker image”
>>
>> *shrugs* I use a different release script after some changes
>> broke the in-tree version for building on OS X and I couldn’t get the fixes
>> committed upstream.  So not sure what the problems are that you are hitting.
>>
>> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu <
>> nvadiv...@hortonworks.com> wrote:
>> >
>> > It will be helpful if we can get the correct steps, and also update the
>> wiki.
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+
>> Release+Validation
>>
>> Yup. Looking forward to seeing it.
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Thank you all very much for the testing, feedback and discussion!

I was able to build outside docker, by following the steps Nanda described,
I saw the same problem; then I tried 3.0.2 released a while back, it has
the same issue.

As Allen pointed out, it seems the steps to build site are not correct. I
have not figured out the correct steps yet.

At this point, I think this issue should not block the 3.0.3 issue. While
at the same time we need to figure out the right steps to build the site.
Would you please let me know if you think differently?

We only have the site build issue reported so far. And we don't have enough
PMC votes yet. So need some more PMCs to help.

Thanks again, and best regards,

--Yongjun


On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer 
wrote:

> > On Jun 7, 2018, at 11:47 AM, Steve Loughran 
> wrote:
> >
> > Actually, Yongjun has been really good at helping me get set up for a
> 2.7.7 release, including "things you need to do to get GPG working in the
> docker image”
>
> *shrugs* I use a different release script after some changes broke
> the in-tree version for building on OS X and I couldn’t get the fixes
> committed upstream.  So not sure what the problems are that you are hitting.
>
> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu <
> nvadiv...@hortonworks.com> wrote:
> >
> > It will be helpful if we can get the correct steps, and also update the
> wiki.
> > https://cwiki.apache.org/confluence/display/HADOOP/
> Hadoop+Release+Validation
>
> Yup. Looking forward to seeing it.
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Thank you all so much for the testing and vote!

Nanda reported the following problem:

The following links in site documentation are broken
- Changelog and Release Notes
- Unix Shell API
- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml

Site documentation was generated using the below steps
*- mvn site:site*
*- mkdir -p /tmp/site && mvn site:stage -DstagingDirectory=/tmp/site*
*- Browse to file:///tmp/site/hadoop-project/index.html.*

Thanks,
Nanda


My build in docker was fine, and when I tested out the build outside
docker, I tested the RC binary tarbal and regular build with source
tarball, but missed out the site:site build.

I tried the steps Nanda did, and I'm even hitting an issue at the "mvn
site:site" which Nanda did not hit.

What I'm seeing is:

[INFO] 

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.6:site
(default-cli) on project hadoop-main: failed to get report for
org.apache.maven.plugins:maven-javadoc-plugin: Failed to execute goal on
project hadoop-hdfs: Could not resolve dependencies for project
org.apache.hadoop:hadoop-hdfs:jar:3.0.3: *Failure to find
org.apache.hadoop:hadoop-hdfs-client:jar:3.0.3
in https://s3-us-west-2.amazonaws.com/dynamodb-local/release
<https://s3-us-west-2.amazonaws.com/dynamodb-local/release>* was cached in
the local repository, resolution will not be reattempted until the update
interval of dynamodb-local-oregon has elapsed or updates are forced ->
[Help 1]
[ERROR]


I wonder if anyone else hit this problem. I was able to build 3.0.2 with
the same env. The build is looking for 3.0.3 artifacts in
*https://s3-us-west-2.amazonaws.com/dynamodb-local/release
<https://s3-us-west-2.amazonaws.com/dynamodb-local/release>, *which I'm
sure is the right thing, because 3.0.3 is not yet released. Or probably I
missed something in my env for this, which I have to investigate.

If anyone can follow Nanda's steps do see different outcome, that would be
interesting. Very much appreciated.

Thanks again!

--Yongjun

On Wed, Jun 6, 2018 at 12:59 AM, Kitti Nánási  wrote:

> Thanks Yongjun for working on this!
>
> +1 (non-binding)
>
> - checked out git tag release-3.0.3-RC0
> - built from source on Mac OS X 10.13.4, java version 1.8.0_171
> - deployed on a 3 node cluster
> - ran terasort, teragen, teravalidate with success
> - executed basic hdfs commands
> - created snapshots and produced snapshot diffs
>
> Thanks,
> Kitti Nanasi
>
>
> On Wed, Jun 6, 2018 at 8:06 AM, Takanobu Asanuma 
> wrote:
>
>> Sorry, I forgot to write my vote in the last message.
>>
>> +1 (non-binding)
>>
>> Regards,
>> Takanobu
>>
>> > -Original Message-----
>> > From: Takanobu Asanuma [mailto:tasan...@yahoo-corp.jp]
>> > Sent: Wednesday, June 06, 2018 10:22 AM
>> > To: Gabor Bota 
>> > Cc: Yongjun Zhang ; nvadiv...@hortonworks.com;
>> > sbaner...@hortonworks.com; Hadoop Common 
>> ;
>> > Hdfs-dev ; mapreduce-...@hadoop.apache.org;
>> > yarn-...@hadoop.apache.org
>> > Subject: RE: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
>> >
>> > Thanks for driving this release, Yongjun!
>> >
>> > - Verified checksums
>> > - Succeeded native package build on CentOS 7
>> > - Started a cluster with 1 master and 5 slaves
>> > - Verified Web UI (NN, RM, JobHistory, Timeline)
>> > - Verified Teragen/Terasort jobs
>> > - Verified some operations for erasure coding
>> >
>> > Regards,
>> > Takanobu
>> >
>> > > -Original Message-
>> > > From: Gabor Bota [mailto:gabor.b...@cloudera.com]
>> > > Sent: Tuesday, June 05, 2018 9:18 PM
>> > > To: nvadiv...@hortonworks.com
>> > > Cc: Yongjun Zhang ; sbaner...@hortonworks.com;
>> > > Hadoop Common ; Hdfs-dev
>> > > ; mapreduce-...@hadoop.apache.org;
>> > > yarn-...@hadoop.apache.org
>> > > Subject: Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
>> > >
>> > > Thanks for the work Yongjun!
>> > >
>> > > +1 (non-binding)
>> > >
>> > >- checked out git tag release-3.0.3-RC0. Thanks for adding this
>> > Yongjun,
>> > >it worked.
>> > >- S3A integration (mvn verify) test run were successful on
>> eu-west-1
>> > >besides of one test issue reported in HADOOP-14927.
>> > >- built from source on Mac OS X 10.13.4, java version
>> 8.0.171-oracle
>> > >- deployed on a 3 node cluster (HDFS HA, Non-HA YARN)
>> > >- verified

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Hi Gabor,

I got the git tag in, it's release-3.0.3-RC0. Would you please give it a
try?

It should correspond to

commit 37fd7d752db73d984dc31e0cdfd590d252f5e075
Author: Yongjun Zhang 
Date:   Wed May 30 00:07:33 2018 -0700

Update version to 3.0.3 to prepare for 3.0.3 release


Thanks,

--Yongjun

On Fri, Jun 1, 2018 at 4:17 AM, Gabor Bota  wrote:

> Hi Yongjun,
>
> Thank you for working on this release. Is there a git tag in the upstream
> repo which can be checked out? I'd like to build the release from source.
>
> Regards,
> Gabor
>
> On Fri, Jun 1, 2018 at 7:57 AM Shashikant Banerjee <
> sbaner...@hortonworks.com> wrote:
>
>> Looks like the link with the filter seems to be private. I can't see the
>> blocker list.
>> https://issues.apache.org/jira/issues/?filter=12343997
>>
>> Meanwhile , I will be working on testing the release.
>>
>> Thanks
>> Shashi
>> On 6/1/18, 11:18 AM, "Yongjun Zhang"  wrote:
>>
>> Greetings all,
>>
>> I've created the first release candidate (RC0) for Apache Hadoop
>> 3.0.3. This is our next maintenance release to follow up 3.0.2. It
>> includes
>> about 249
>> important fixes and improvements, among which there are 8 blockers.
>> See
>> https://issues.apache.org/jira/issues/?filter=12343997
>>
>> The RC artifacts are available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>>
>> The maven artifacts are available via
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1126
>>
>> Please try the release and vote; the vote will run for the usual 5
>> working
>> days, ending on 06/07/2018 PST time. Would really appreciate your
>> participation here.
>>
>> I bumped into quite some issues along the way, many thanks to quite a
>> few
>> people who helped, especially Sammi Chen, Andrew Wang, Junping Du,
>> Eddy Xu.
>>
>> Thanks,
>>
>> --Yongjun
>>
>>
>>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Many thanks Shashikant, Zsolt, and Ajay!

Best,

--Yongjun

On Sat, Jun 2, 2018 at 7:34 AM, Shashikant Banerjee <
sbaner...@hortonworks.com> wrote:

> Thanks for working on this Yongjun!
>
> +1 (non-binding)
>  - verified signatures and checksums
>  - built from source and setup single node cluster
>  - ran basic hdfs operations
> - basic sanity check of NN
>
> Thanks
> Shashi
>
> On 6/2/18, 7:31 PM, "Zsolt Venczel"  wrote:
>
> Thanks Yongjun for working on this!
>
> +1 (non-binding)
>
>  - built from source with native library
>  - run hadoop-hdfs-client tests with native library that all passed
>  - set up a cluster with 3 nodes and run teragen, terasort and
> teravalidate
> successfully
>  - created two snapshots and produced a snapshot diff successfully
>  - checked out the webui, looked at the file structure and double
> checked
> the created snapshots
>
> Thanks and best regards,
> Zsolt
>
> On Sat, Jun 2, 2018 at 11:33 AM Ajay Kumar  >
> wrote:
>
> > Thanks for working on this Yongjun!!
> >
> > +1 (non-binding)
> > - verified signatures and checksums
> > - built from source and setup single node cluster
> > - ran basic hdfs operations
> > - ran TestDFSIO(read/write), wordcount, pi jobs.
> > - basic sanity check of NN, RM UI
> >
> > Thanks,
> > Ajay
> >
> > On 6/2/18, 12:45 AM, "Shashikant Banerjee" <
> sbaner...@hortonworks.com>
> > wrote:
> >
> > Hi Yongjun,
> >
> > I am able to see the list after logging in now.
> >
> > Thanks
> > Shashi
> >
> > From: Yongjun Zhang 
> > Date: Friday, June 1, 2018 at 9:11 PM
> > To: Gabor Bota 
> > Cc: Shashikant Banerjee , Hadoop
> Common <
> > common-dev@hadoop.apache.org>, Hdfs-dev ,
> "
> > mapreduce-...@hadoop.apache.org" ,
> "
> > yarn-...@hadoop.apache.org" 
> > Subject: Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
> >
> >
> > Thanks for the feedback!
> >
> > Hi Shashikant, I thought I have made the filter visible to jira
> users,
> > now I changed it to be visible to all logged-in users of jira.
> Please let
> > me know if you can not see it.
> >
> > Hi Gabor,
> >
> > Good question. I forgot to mention, I have tried to add tag
> earlier,
> > as step 7,8 9 in
> > https://wiki.apache.org/hadoop/HowToRelease, but these steps
> seem to
> > not push anything to git. I suspect step 4 should have been run with
> > --rc-label , and checked with Andrew, he said it doesn't matter and
> often
> > people don't use the rc label.
> >
> > I probably should mention that the build is on commit id
> > 37fd7d752db73d984dc31e0cdfd590d252f5e075.
> >
> > The source is also available at
> > https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
> >
> > Thanks.
> >
> > --Yongjun
> >
> > On Fri, Jun 1, 2018 at 4:17 AM, Gabor Bota <
> gabor.b...@cloudera.com
> > <mailto:gabor.b...@cloudera.com>> wrote:
> > Hi Yongjun,
> >
> > Thank you for working on this release. Is there a git tag in the
> > upstream repo which can be checked out? I'd like to build the
> release from
> > source.
> >
> > Regards,
> > Gabor
> >
> > On Fri, Jun 1, 2018 at 7:57 AM Shashikant Banerjee <
> > sbaner...@hortonworks.com<mailto:sbaner...@hortonworks.com>> wrote:
> > Looks like the link with the filter seems to be private. I can't
> see
> > the blocker list.
> > https://issues.apache.org/jira/issues/?filter=12343997
> >
> > Meanwhile , I will be working on testing the release.
> >
> > Thanks
> > Shashi
> > On 6/1/18, 11:18 AM, "Yongjun Zhang"   > yjzhan...@apache.org>> wrote:
> >
> > Greetings all,
> >
> > I've created the first release candidate (RC0) for Apache
> Hadoop
> > 3.0.3. This is our next maintenance release to follow up
> 3.0.2. It
> > includes
> > about 249
&g

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Hi Eric,

Thanks so much for your work and quick feedback, that's cool!

--Yongjun

On Fri, Jun 1, 2018 at 10:25 AM, Eric Payne <
eric.payne1...@yahoo.com.invalid> wrote:

>
>
> Thanks a lot, Yongjun, for your hard work on this release.
>
> +1
> - Built from source
> - Installed on 6 node pseudo cluster
>
>
> Tested the following in the Capacity Scheduler:
> - Verified that running apps in labelled queues restricts tasks to the
> labelled nodes.
> - Verified that various queue config properties for CS are refreshable
> - Verified streaming jobs work as expected
> - Verified that user weights work as expected
> - Verified that FairOrderingPolicy in a CS queue will evenly assign
> resources
> - Verified running yarn shell application runs as expected
>
>
>
>
>
>
>
> On Friday, June 1, 2018, 12:48:26 AM CDT, Yongjun Zhang <
> yjzhan...@apache.org> wrote:
>
>
>
>
>
> Greetings all,
>
> I've created the first release candidate (RC0) for Apache Hadoop
> 3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
> about 249
> important fixes and improvements, among which there are 8 blockers. See
> https://issues.apache.org/jira/issues/?filter=12343997
>
> The RC artifacts are available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>
> The maven artifacts are available via
> https://repository.apache.org/content/repositories/orgapachehadoop-1126
>
> Please try the release and vote; the vote will run for the usual 5 working
> days, ending on 06/07/2018 PST time. Would really appreciate your
> participation here.
>
> I bumped into quite some issues along the way, many thanks to quite a few
> people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.
>
> Thanks,
>
> --Yongjun
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Hi Rushabh,

Would you please try again?

Permissions

This filter is visible to:

   - *Group*: all-developers
   - *Group*: jira-users


Thanks.

--Yongjun

On Fri, Jun 1, 2018 at 8:48 AM, Rushabh Shah  wrote:

> Hi Yongjun,
> Thanks for all the effort you put in for this release.
>
>> https://issues.apache.org/jira/issues/?filter=12343997
>
>  Even after logging in, I am not able to see any jiras.
> It says "The requested filter doesn't exist or is private."
>
> On Fri, Jun 1, 2018 at 10:41 AM, Yongjun Zhang 
> wrote:
>
>> Thanks for the feedback!
>>
>> Hi Shashikant, I thought I have made the filter visible to jira users, now
>> I changed it to be visible to all logged-in users of jira. Please let me
>> know if you can not see it.
>>
>> Hi Gabor,
>>
>> Good question. I forgot to mention, I have tried to add tag earlier, as
>> step 7,8 9 in
>> https://wiki.apache.org/hadoop/HowToRelease, but these steps seem to not
>> push anything to git. I suspect step 4 should have been run with
>> *--rc-label* , and checked with Andrew, he said it doesn't matter and
>> often
>>
>> people don't use the rc label.
>>
>> I probably should mention that the build is on commit id
>> 37fd7d752db73d984dc31e0cdfd590d252f5e075.
>>
>> The source is also available at   https://dist.apache.org/repos/
>> dist/dev/hadoop/3.0.3-RC0/
>>
>> Thanks.
>>
>> --Yongjun
>>
>> On Fri, Jun 1, 2018 at 4:17 AM, Gabor Bota 
>> wrote:
>>
>> > Hi Yongjun,
>> >
>> > Thank you for working on this release. Is there a git tag in the
>> upstream
>> > repo which can be checked out? I'd like to build the release from
>> source.
>> >
>> > Regards,
>> > Gabor
>> >
>> > On Fri, Jun 1, 2018 at 7:57 AM Shashikant Banerjee <
>> > sbaner...@hortonworks.com> wrote:
>> >
>> >> Looks like the link with the filter seems to be private. I can't see
>> the
>> >> blocker list.
>> >> https://issues.apache.org/jira/issues/?filter=12343997
>> >>
>> >> Meanwhile , I will be working on testing the release.
>> >>
>> >> Thanks
>> >> Shashi
>> >> On 6/1/18, 11:18 AM, "Yongjun Zhang"  wrote:
>> >>
>> >> Greetings all,
>> >>
>> >> I've created the first release candidate (RC0) for Apache Hadoop
>> >> 3.0.3. This is our next maintenance release to follow up 3.0.2. It
>> >> includes
>> >> about 249
>> >> important fixes and improvements, among which there are 8 blockers.
>> >> See
>> >> https://issues.apache.org/jira/issues/?filter=12343997
>> >>
>> >> The RC artifacts are available at:
>> >> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>> >>
>> >> The maven artifacts are available via
>> >> https://repository.apache.org/content/repositories/
>> >> orgapachehadoop-1126
>> >>
>> >> Please try the release and vote; the vote will run for the usual 5
>> >> working
>> >> days, ending on 06/07/2018 PST time. Would really appreciate your
>> >> participation here.
>> >>
>> >> I bumped into quite some issues along the way, many thanks to
>> quite a
>> >> few
>> >> people who helped, especially Sammi Chen, Andrew Wang, Junping Du,
>> >> Eddy Xu.
>> >>
>> >> Thanks,
>> >>
>> >> --Yongjun
>> >>
>> >>
>> >>
>>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Thanks for the feedback!

Hi Shashikant, I thought I have made the filter visible to jira users, now
I changed it to be visible to all logged-in users of jira. Please let me
know if you can not see it.

Hi Gabor,

Good question. I forgot to mention, I have tried to add tag earlier, as
step 7,8 9 in
https://wiki.apache.org/hadoop/HowToRelease, but these steps seem to not
push anything to git. I suspect step 4 should have been run with
*--rc-label* , and checked with Andrew, he said it doesn't matter and often
people don't use the rc label.

I probably should mention that the build is on commit id
37fd7d752db73d984dc31e0cdfd590d252f5e075.

The source is also available at   https://dist.apache.org/repos/
dist/dev/hadoop/3.0.3-RC0/

Thanks.

--Yongjun

On Fri, Jun 1, 2018 at 4:17 AM, Gabor Bota  wrote:

> Hi Yongjun,
>
> Thank you for working on this release. Is there a git tag in the upstream
> repo which can be checked out? I'd like to build the release from source.
>
> Regards,
> Gabor
>
> On Fri, Jun 1, 2018 at 7:57 AM Shashikant Banerjee <
> sbaner...@hortonworks.com> wrote:
>
>> Looks like the link with the filter seems to be private. I can't see the
>> blocker list.
>> https://issues.apache.org/jira/issues/?filter=12343997
>>
>> Meanwhile , I will be working on testing the release.
>>
>> Thanks
>> Shashi
>> On 6/1/18, 11:18 AM, "Yongjun Zhang"  wrote:
>>
>> Greetings all,
>>
>> I've created the first release candidate (RC0) for Apache Hadoop
>> 3.0.3. This is our next maintenance release to follow up 3.0.2. It
>> includes
>> about 249
>> important fixes and improvements, among which there are 8 blockers.
>> See
>> https://issues.apache.org/jira/issues/?filter=12343997
>>
>> The RC artifacts are available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
>>
>> The maven artifacts are available via
>> https://repository.apache.org/content/repositories/
>> orgapachehadoop-1126
>>
>> Please try the release and vote; the vote will run for the usual 5
>> working
>> days, ending on 06/07/2018 PST time. Would really appreciate your
>> participation here.
>>
>> I bumped into quite some issues along the way, many thanks to quite a
>> few
>> people who helped, especially Sammi Chen, Andrew Wang, Junping Du,
>> Eddy Xu.
>>
>> Thanks,
>>
>> --Yongjun
>>
>>
>>

[VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Yongjun Zhang]

Greetings all,

I've created the first release candidate (RC0) for Apache Hadoop
3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
about 249
important fixes and improvements, among which there are 8 blockers. See
https://issues.apache.org/jira/issues/?filter=12343997

The RC artifacts are available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/

The maven artifacts are available via
https://repository.apache.org/content/repositories/orgapachehadoop-1126

Please try the release and vote; the vote will run for the usual 5 working
days, ending on 06/07/2018 PST time. Would really appreciate your
participation here.

I bumped into quite some issues along the way, many thanks to quite a few
people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.

Thanks,

--Yongjun

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Unfortunately still hit another issue, step 4 in
https://wiki.apache.org/hadoop/HowToRelease doesn't deploy artifacts to
Nexus. According to Andrew, it should and we should skip step 10.

Though I have the release tarballs, because of this deployment issue, I
still have to investigate further. Sorry for the inconvenience.

--Yongjun

On Wed, May 30, 2018 at 11:49 AM, Yongjun Zhang 
wrote:

> Hi,
>
> The build issues are all solved, and I have cut the 3.0.3 branch and close
> to get a build out. Since it's taking me a bit more time (I expect to send
> vote invitation email by today). I would like to send a heads-up notice now.
>
> Thank you all for feedback, and many thanks to Sammi Chen, Andrew Wang,
> Eddy Xu who helped when I tried to solve the build issues.
>
> At this point, please be aware of the existence of branch-3.0,
> branch-3.0.3.
>
> Best,
>
> --Yongjun
>
>
>
> On Sat, May 26, 2018 at 11:52 PM, Yongjun Zhang 
> wrote:
>
>> HI,
>>
>> I did build before cut branch and hit some issues, have not got to the
>> bottom, will cut branch after the build issues are resolved.
>>
>> Thanks.
>>
>> --Yongjun
>>
>> On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang 
>> wrote:
>>
>>> Hi All,
>>>
>>> I will be working on cutting the 3.0.3 branch and trying a build today.
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>>
>>> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang 
>>> wrote:
>>>
>>>> Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
>>>> sooner.
>>>>
>>>> --Yongjun
>>>>
>>>>
>>>> On Wed, May 23, 2018 at 2:18 PM, Eric Badger  wrote:
>>>>
>>>>> My thinking is to cut the branch in next couple of days and create RC
>>>>> for
>>>>> vote at the end of month.
>>>>>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th)
>>>>> and vote for RC on May 30th
>>>>>   I much prefer to wait to cut the branch until just before the
>>>>> production of the release and the vote. With so many branches, we 
>>>>> sometimes
>>>>> miss putting critical bug fixes in unreleased branches if thebranch is
>>>>> cut too early.
>>>>>
>>>>> Echoing Eric Payne, I think we should wait to cut the branch until we
>>>>> are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
>>>>> is to be on May 30).
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang 
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have gardened the jiras for 3.0.3, and have the following open
>>>>>> issues:
>>>>>>
>>>>>> https://issues.apache.org/jira/issues/?filter=12343970
>>>>>>
>>>>>> Two of them are blockers, one of them (YARN-8346) has already got +1
>>>>>> for
>>>>>> patch, the other (YARN-8108) will take longer time to resolve and it
>>>>>> seems
>>>>>> we can possibly push it to next release given 3.0.2 also has the
>>>>>> issue.
>>>>>>
>>>>>> My thinking is to cut the branch in next couple of days and create RC
>>>>>> for
>>>>>> vote at the end of month.
>>>>>>
>>>>>> Comments are welcome.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> --Yongjun
>>>>>>
>>>>>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C 
>>>>>> wrote:
>>>>>>
>>>>>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a
>>>>>> fix that
>>>>>> > will enable HBase to use Hadoop 3.0.x in the production line.
>>>>>> >
>>>>>> > thanks
>>>>>> > Vrushali
>>>>>> >
>>>>>> >
>>>>>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang >>>>> >
>>>>>> > wrote:
>>>>>> >
>>>>>> >> Thanks Wei-Chiu and Haibo for the feedback!
>>>>>> >>
>>>>>> >> Good thing is that I have 

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Sorry, forgot to mention Junping Du, who also helped me a lot. Many thanks
Junping!

Thanks.

--Yongjun

On Wed, May 30, 2018 at 11:49 AM, Yongjun Zhang 
wrote:

> Hi,
>
> The build issues are all solved, and I have cut the 3.0.3 branch and close
> to get a build out. Since it's taking me a bit more time (I expect to send
> vote invitation email by today). I would like to send a heads-up notice now.
>
> Thank you all for feedback, and many thanks to Sammi Chen, Andrew Wang,
> Eddy Xu who helped when I tried to solve the build issues.
>
> At this point, please be aware of the existence of branch-3.0,
> branch-3.0.3.
>
> Best,
>
> --Yongjun
>
>
>
> On Sat, May 26, 2018 at 11:52 PM, Yongjun Zhang 
> wrote:
>
>> HI,
>>
>> I did build before cut branch and hit some issues, have not got to the
>> bottom, will cut branch after the build issues are resolved.
>>
>> Thanks.
>>
>> --Yongjun
>>
>> On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang 
>> wrote:
>>
>>> Hi All,
>>>
>>> I will be working on cutting the 3.0.3 branch and trying a build today.
>>>
>>> Thanks.
>>>
>>> --Yongjun
>>>
>>>
>>>
>>> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang 
>>> wrote:
>>>
>>>> Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
>>>> sooner.
>>>>
>>>> --Yongjun
>>>>
>>>>
>>>> On Wed, May 23, 2018 at 2:18 PM, Eric Badger  wrote:
>>>>
>>>>> My thinking is to cut the branch in next couple of days and create RC
>>>>> for
>>>>> vote at the end of month.
>>>>>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th)
>>>>> and vote for RC on May 30th
>>>>>   I much prefer to wait to cut the branch until just before the
>>>>> production of the release and the vote. With so many branches, we 
>>>>> sometimes
>>>>> miss putting critical bug fixes in unreleased branches if thebranch is
>>>>> cut too early.
>>>>>
>>>>> Echoing Eric Payne, I think we should wait to cut the branch until we
>>>>> are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
>>>>> is to be on May 30).
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang 
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have gardened the jiras for 3.0.3, and have the following open
>>>>>> issues:
>>>>>>
>>>>>> https://issues.apache.org/jira/issues/?filter=12343970
>>>>>>
>>>>>> Two of them are blockers, one of them (YARN-8346) has already got +1
>>>>>> for
>>>>>> patch, the other (YARN-8108) will take longer time to resolve and it
>>>>>> seems
>>>>>> we can possibly push it to next release given 3.0.2 also has the
>>>>>> issue.
>>>>>>
>>>>>> My thinking is to cut the branch in next couple of days and create RC
>>>>>> for
>>>>>> vote at the end of month.
>>>>>>
>>>>>> Comments are welcome.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> --Yongjun
>>>>>>
>>>>>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C 
>>>>>> wrote:
>>>>>>
>>>>>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a
>>>>>> fix that
>>>>>> > will enable HBase to use Hadoop 3.0.x in the production line.
>>>>>> >
>>>>>> > thanks
>>>>>> > Vrushali
>>>>>> >
>>>>>> >
>>>>>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang >>>>> >
>>>>>> > wrote:
>>>>>> >
>>>>>> >> Thanks Wei-Chiu and Haibo for the feedback!
>>>>>> >>
>>>>>> >> Good thing is that I have made the following note couple of days
>>>>>> ago when
>>>>>> >> I
>>>>>> >> looked the at branch diff, so we are on the same page:
>>>>>> >>
>

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Hi,

The build issues are all solved, and I have cut the 3.0.3 branch and close
to get a build out. Since it's taking me a bit more time (I expect to send
vote invitation email by today). I would like to send a heads-up notice now.

Thank you all for feedback, and many thanks to Sammi Chen, Andrew Wang,
Eddy Xu who helped when I tried to solve the build issues.

At this point, please be aware of the existence of branch-3.0, branch-3.0.3.

Best,

--Yongjun



On Sat, May 26, 2018 at 11:52 PM, Yongjun Zhang  wrote:

> HI,
>
> I did build before cut branch and hit some issues, have not got to the
> bottom, will cut branch after the build issues are resolved.
>
> Thanks.
>
> --Yongjun
>
> On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang 
> wrote:
>
>> Hi All,
>>
>> I will be working on cutting the 3.0.3 branch and trying a build today.
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>>
>> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang 
>> wrote:
>>
>>> Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
>>> sooner.
>>>
>>> --Yongjun
>>>
>>>
>>> On Wed, May 23, 2018 at 2:18 PM, Eric Badger  wrote:
>>>
>>>> My thinking is to cut the branch in next couple of days and create RC
>>>> for
>>>> vote at the end of month.
>>>>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
>>>> vote for RC on May 30th
>>>>   I much prefer to wait to cut the branch until just before the
>>>> production of the release and the vote. With so many branches, we sometimes
>>>> miss putting critical bug fixes in unreleased branches if the    branch is
>>>> cut too early.
>>>>
>>>> Echoing Eric Payne, I think we should wait to cut the branch until we
>>>> are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
>>>> is to be on May 30).
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>>> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have gardened the jiras for 3.0.3, and have the following open
>>>>> issues:
>>>>>
>>>>> https://issues.apache.org/jira/issues/?filter=12343970
>>>>>
>>>>> Two of them are blockers, one of them (YARN-8346) has already got +1
>>>>> for
>>>>> patch, the other (YARN-8108) will take longer time to resolve and it
>>>>> seems
>>>>> we can possibly push it to next release given 3.0.2 also has the issue.
>>>>>
>>>>> My thinking is to cut the branch in next couple of days and create RC
>>>>> for
>>>>> vote at the end of month.
>>>>>
>>>>> Comments are welcome.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> --Yongjun
>>>>>
>>>>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C 
>>>>> wrote:
>>>>>
>>>>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a fix
>>>>> that
>>>>> > will enable HBase to use Hadoop 3.0.x in the production line.
>>>>> >
>>>>> > thanks
>>>>> > Vrushali
>>>>> >
>>>>> >
>>>>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang 
>>>>> > wrote:
>>>>> >
>>>>> >> Thanks Wei-Chiu and Haibo for the feedback!
>>>>> >>
>>>>> >> Good thing is that I have made the following note couple of days
>>>>> ago when
>>>>> >> I
>>>>> >> looked the at branch diff, so we are on the same page:
>>>>> >>
>>>>> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x
>>>>> gets TSv2
>>>>> >> related hbase jars, not the user classpath. Contributed by Varun
>>>>> Saxena."
>>>>> >>
>>>>> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
>>>>> comment
>>>>> >> below:
>>>>> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>>>>> >> ntId=16457649&
>>>>> >> page=com.atlassian.jira.plugin.system.issuetabpanels
>>>>> >> <https://is

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

HI,

I did build before cut branch and hit some issues, have not got to the
bottom, will cut branch after the build issues are resolved.

Thanks.

--Yongjun

On Sat, May 26, 2018 at 1:46 PM, Yongjun Zhang <yjzhan...@apache.org> wrote:

> Hi All,
>
> I will be working on cutting the 3.0.3 branch and trying a build today.
>
> Thanks.
>
> --Yongjun
>
>
>
> On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang <yzh...@cloudera.com>
> wrote:
>
>> Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
>> sooner.
>>
>> --Yongjun
>>
>>
>> On Wed, May 23, 2018 at 2:18 PM, Eric Badger <ebad...@oath.com> wrote:
>>
>>> My thinking is to cut the branch in next couple of days and create RC for
>>> vote at the end of month.
>>>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
>>> vote for RC on May 30th
>>>   I much prefer to wait to cut the branch until just before the
>>> production of the release and the vote. With so many branches, we sometimes
>>> miss putting critical bug fixes in unreleased branches if thebranch is
>>> cut too early.
>>>
>>> Echoing Eric Payne, I think we should wait to cut the branch until we
>>> are actually creating the RC to vote on (i.e. on May 29 or 30 if the vote
>>> is to be on May 30).
>>>
>>> Eric
>>>
>>>
>>>
>>> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang <yzh...@cloudera.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have gardened the jiras for 3.0.3, and have the following open issues:
>>>>
>>>> https://issues.apache.org/jira/issues/?filter=12343970
>>>>
>>>> Two of them are blockers, one of them (YARN-8346) has already got +1 for
>>>> patch, the other (YARN-8108) will take longer time to resolve and it
>>>> seems
>>>> we can possibly push it to next release given 3.0.2 also has the issue.
>>>>
>>>> My thinking is to cut the branch in next couple of days and create RC
>>>> for
>>>> vote at the end of month.
>>>>
>>>> Comments are welcome.
>>>>
>>>> Thanks,
>>>>
>>>> --Yongjun
>>>>
>>>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C <vrushalic2...@gmail.com>
>>>> wrote:
>>>>
>>>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a fix
>>>> that
>>>> > will enable HBase to use Hadoop 3.0.x in the production line.
>>>> >
>>>> > thanks
>>>> > Vrushali
>>>> >
>>>> >
>>>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang <yzh...@cloudera.com>
>>>> > wrote:
>>>> >
>>>> >> Thanks Wei-Chiu and Haibo for the feedback!
>>>> >>
>>>> >> Good thing is that I have made the following note couple of days ago
>>>> when
>>>> >> I
>>>> >> looked the at branch diff, so we are on the same page:
>>>> >>
>>>> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x gets
>>>> TSv2
>>>> >> related hbase jars, not the user classpath. Contributed by Varun
>>>> Saxena."
>>>> >>
>>>> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
>>>> comment
>>>> >> below:
>>>> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>>>> >> ntId=16457649&
>>>> >> page=com.atlassian.jira.plugin.system.issuetabpanels
>>>> >> <https://issues.apache.org/jira/browse/YARN-7190?focusedComm
>>>> entId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>>>>
>>>> >> :
>>>> >> comment-tabpanel#comment-16457649
>>>> >>
>>>> >>
>>>> >> In addition, I will revert   https://issues.apache.org/
>>>> >> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>>>> >>
>>>> >>
>>>> >> Best,
>>>> >>
>>>> >> --Yongjun
>>>> >>
>>>> >> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen <haiboc...@cloudera.com>
>>>> >> wrote:
>>>> >>
>>>> >> > +1 on adding YARN-7190 to Hadoo

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Hi All,

I will be working on cutting the 3.0.3 branch and trying a build today.

Thanks.

--Yongjun



On Wed, May 23, 2018 at 3:31 PM, Yongjun Zhang <yzh...@cloudera.com> wrote:

> Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
> sooner.
>
> --Yongjun
>
>
> On Wed, May 23, 2018 at 2:18 PM, Eric Badger <ebad...@oath.com> wrote:
>
>> My thinking is to cut the branch in next couple of days and create RC for
>> vote at the end of month.
>>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
>> vote for RC on May 30th
>>   I much prefer to wait to cut the branch until just before the
>> production of the release and the vote. With so many branches, we sometimes
>> miss putting critical bug fixes in unreleased branches if thebranch is
>> cut too early.
>>
>> Echoing Eric Payne, I think we should wait to cut the branch until we are
>> actually creating the RC to vote on (i.e. on May 29 or 30 if the vote is to
>> be on May 30).
>>
>> Eric
>>
>>
>>
>> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang <yzh...@cloudera.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have gardened the jiras for 3.0.3, and have the following open issues:
>>>
>>> https://issues.apache.org/jira/issues/?filter=12343970
>>>
>>> Two of them are blockers, one of them (YARN-8346) has already got +1 for
>>> patch, the other (YARN-8108) will take longer time to resolve and it
>>> seems
>>> we can possibly push it to next release given 3.0.2 also has the issue.
>>>
>>> My thinking is to cut the branch in next couple of days and create RC for
>>> vote at the end of month.
>>>
>>> Comments are welcome.
>>>
>>> Thanks,
>>>
>>> --Yongjun
>>>
>>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C <vrushalic2...@gmail.com>
>>> wrote:
>>>
>>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a fix
>>> that
>>> > will enable HBase to use Hadoop 3.0.x in the production line.
>>> >
>>> > thanks
>>> > Vrushali
>>> >
>>> >
>>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang <yzh...@cloudera.com>
>>> > wrote:
>>> >
>>> >> Thanks Wei-Chiu and Haibo for the feedback!
>>> >>
>>> >> Good thing is that I have made the following note couple of days ago
>>> when
>>> >> I
>>> >> looked the at branch diff, so we are on the same page:
>>> >>
>>> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x gets
>>> TSv2
>>> >> related hbase jars, not the user classpath. Contributed by Varun
>>> Saxena."
>>> >>
>>> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
>>> comment
>>> >> below:
>>> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>>> >> ntId=16457649&
>>> >> page=com.atlassian.jira.plugin.system.issuetabpanels
>>> >> <https://issues.apache.org/jira/browse/YARN-7190?focusedComm
>>> entId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>>>
>>> >> :
>>> >> comment-tabpanel#comment-16457649
>>> >>
>>> >>
>>> >> In addition, I will revert   https://issues.apache.org/
>>> >> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>>> >>
>>> >>
>>> >> Best,
>>> >>
>>> >> --Yongjun
>>> >>
>>> >> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen <haiboc...@cloudera.com>
>>> >> wrote:
>>> >>
>>> >> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it is
>>> >> > technically incompatible.
>>> >> > It is critical enough to justify being an exception, IMO.
>>> >> >
>>> >> > Added Rohith and Vrushali
>>> >> >
>>> >> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <weic...@apache.org
>>> >
>>> >> > wrote:
>>> >> >
>>> >> >> Thanks Yongjun for driving 3.0.3 release!
>>> >> >>
>>> >> >> IMHO, could we consider adding YARN-7190
>>> >> >> <https://issues.apache.org/jira/browse/YARN-7190> 

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Thanks Eric. Sounds good. I may try to see if I can do the branching/RC
sooner.

--Yongjun

On Wed, May 23, 2018 at 2:18 PM, Eric Badger <ebad...@oath.com> wrote:

> My thinking is to cut the branch in next couple of days and create RC for
> vote at the end of month.
>   >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
> vote for RC on May 30th
>   I much prefer to wait to cut the branch until just before the production
> of the release and the vote. With so many branches, we sometimes miss
> putting critical bug fixes in unreleased branches if thebranch is cut
> too early.
>
> Echoing Eric Payne, I think we should wait to cut the branch until we are
> actually creating the RC to vote on (i.e. on May 29 or 30 if the vote is to
> be on May 30).
>
> Eric
>
>
>
> On Wed, May 23, 2018 at 4:11 PM, Yongjun Zhang <yzh...@cloudera.com>
> wrote:
>
>> Hi,
>>
>> I have gardened the jiras for 3.0.3, and have the following open issues:
>>
>> https://issues.apache.org/jira/issues/?filter=12343970
>>
>> Two of them are blockers, one of them (YARN-8346) has already got +1 for
>> patch, the other (YARN-8108) will take longer time to resolve and it seems
>> we can possibly push it to next release given 3.0.2 also has the issue.
>>
>> My thinking is to cut the branch in next couple of days and create RC for
>> vote at the end of month.
>>
>> Comments are welcome.
>>
>> Thanks,
>>
>> --Yongjun
>>
>> On Tue, May 8, 2018 at 11:40 AM, Vrushali C <vrushalic2...@gmail.com>
>> wrote:
>>
>> > +1 for including the YARN-7190 patch in 3.0.3 release. This is a fix
>> that
>> > will enable HBase to use Hadoop 3.0.x in the production line.
>> >
>> > thanks
>> > Vrushali
>> >
>> >
>> > On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang <yzh...@cloudera.com>
>> > wrote:
>> >
>> >> Thanks Wei-Chiu and Haibo for the feedback!
>> >>
>> >> Good thing is that I have made the following note couple of days ago
>> when
>> >> I
>> >> looked the at branch diff, so we are on the same page:
>> >>
>> >>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x gets
>> TSv2
>> >> related hbase jars, not the user classpath. Contributed by Varun
>> Saxena."
>> >>
>> >> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the
>> comment
>> >> below:
>> >> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>> >> ntId=16457649&
>> >> page=com.atlassian.jira.plugin.system.issuetabpanels
>> >> <https://issues.apache.org/jira/browse/YARN-7190?focusedComm
>> entId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>>
>> >> :
>> >> comment-tabpanel#comment-16457649
>> >>
>> >>
>> >> In addition, I will revert   https://issues.apache.org/
>> >> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>> >>
>> >>
>> >> Best,
>> >>
>> >> --Yongjun
>> >>
>> >> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen <haiboc...@cloudera.com>
>> >> wrote:
>> >>
>> >> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it is
>> >> > technically incompatible.
>> >> > It is critical enough to justify being an exception, IMO.
>> >> >
>> >> > Added Rohith and Vrushali
>> >> >
>> >> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <weic...@apache.org>
>> >> > wrote:
>> >> >
>> >> >> Thanks Yongjun for driving 3.0.3 release!
>> >> >>
>> >> >> IMHO, could we consider adding YARN-7190
>> >> >> <https://issues.apache.org/jira/browse/YARN-7190> into the list?
>> >> >> I understand that it is listed as an incompatible change, however,
>> >> because
>> >> >> of this bug, HBase considers the entire Hadoop 3.0.x line not
>> >> production
>> >> >> ready. I feel there's not much point releasing any more 3.0.x
>> releases
>> >> if
>> >> >> downstream projects can't pick it up (after the fact that HBase is
>> one
>> >> of
>> >> >> the most important projects around Hadoop).
>> >> >>
>> >> >> On Mon, May 7,

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Hi,

I have gardened the jiras for 3.0.3, and have the following open issues:

https://issues.apache.org/jira/issues/?filter=12343970

Two of them are blockers, one of them (YARN-8346) has already got +1 for
patch, the other (YARN-8108) will take longer time to resolve and it seems
we can possibly push it to next release given 3.0.2 also has the issue.

My thinking is to cut the branch in next couple of days and create RC for
vote at the end of month.

Comments are welcome.

Thanks,

--Yongjun

On Tue, May 8, 2018 at 11:40 AM, Vrushali C <vrushalic2...@gmail.com> wrote:

> +1 for including the YARN-7190 patch in 3.0.3 release. This is a fix that
> will enable HBase to use Hadoop 3.0.x in the production line.
>
> thanks
> Vrushali
>
>
> On Tue, May 8, 2018 at 10:24 AM, Yongjun Zhang <yzh...@cloudera.com>
> wrote:
>
>> Thanks Wei-Chiu and Haibo for the feedback!
>>
>> Good thing is that I have made the following note couple of days ago when
>> I
>> looked the at branch diff, so we are on the same page:
>>
>>  496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x gets TSv2
>> related hbase jars, not the user classpath. Contributed by Varun Saxena."
>>
>> *YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the comment
>> below:
>> https://issues.apache.org/jira/browse/YARN-7190?focusedComme
>> ntId=16457649&
>> page=com.atlassian.jira.plugin.system.issuetabpanels
>> <https://issues.apache.org/jira/browse/YARN-7190?focusedCommentId=16457649=com.atlassian.jira.plugin.system.issuetabpanels>
>> :
>> comment-tabpanel#comment-16457649
>>
>>
>> In addition, I will revert   https://issues.apache.org/
>> jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.
>>
>>
>> Best,
>>
>> --Yongjun
>>
>> On Tue, May 8, 2018 at 8:57 AM, Haibo Chen <haiboc...@cloudera.com>
>> wrote:
>>
>> > +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it is
>> > technically incompatible.
>> > It is critical enough to justify being an exception, IMO.
>> >
>> > Added Rohith and Vrushali
>> >
>> > On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <weic...@apache.org>
>> > wrote:
>> >
>> >> Thanks Yongjun for driving 3.0.3 release!
>> >>
>> >> IMHO, could we consider adding YARN-7190
>> >> <https://issues.apache.org/jira/browse/YARN-7190> into the list?
>> >> I understand that it is listed as an incompatible change, however,
>> because
>> >> of this bug, HBase considers the entire Hadoop 3.0.x line not
>> production
>> >> ready. I feel there's not much point releasing any more 3.0.x releases
>> if
>> >> downstream projects can't pick it up (after the fact that HBase is one
>> of
>> >> the most important projects around Hadoop).
>> >>
>> >> On Mon, May 7, 2018 at 1:19 PM, Yongjun Zhang <yzh...@cloudera.com>
>> >> wrote:
>> >>
>> >> > Hi Eric,
>> >> >
>> >> > Thanks for the feedback, good point. I will try to clean up things,
>> then
>> >> > cut branch before the release production and vote.
>> >> >
>> >> > Best,
>> >> >
>> >> > --Yongjun
>> >> >
>> >> > On Mon, May 7, 2018 at 8:39 AM, Eric Payne <eric.payne1...@yahoo.com
>> .
>> >> > invalid
>> >> > > wrote:
>> >> >
>> >> > > >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th)
>> and
>> >> vote
>> >> > > for RC on May 30th
>> >> > > I much prefer to wait to cut the branch until just before the
>> >> production
>> >> > > of the release and the vote. With so many branches, we sometimes
>> miss
>> >> > > putting critical bug fixes in unreleased branches if the branch is
>> cut
>> >> > too
>> >> > > early.
>> >> > >
>> >> > > My 2 cents...
>> >> > > Thanks,
>> >> > > -Eric Payne
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > On Monday, May 7, 2018, 12:09:00 AM CDT, Yongjun Zhang <
>> >> > > yjzhan...@apache.org> wrote:
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > &g

Re: [ANNOUNCE] Apache Hadoop 2.9.1 Release

[Yongjun Zhang]

Thanks for the effort doing the release Sammi, very nice work!

--Yongjun

On Tue, May 15, 2018 at 10:21 AM, Chen, Sammi <sammi.c...@intel.com> wrote:

> Hello everyone,
>
> I am glad to announce that Apache Hadoop 2.9.1 has been released.
>
> Apache Hadoop 2.9.1 is the next release of Apache Hadoop 2.9 line. It
> includes 208 bug fixes, improvements and enhancements since previous Apache
> Hadoop 2.9.0 release.
>
>  - For major changes included in Hadoop 2.9 line, please refer to Hadoop
> 2.9.1 main page [1].
>  - For more details about fixes and improvements in 2.9.1 release, please
> refer to CHANGES [2] and RELEASENOTES [3].
>  - For download, please got to download page[4]
>
> Thank you all for contributing to the Apache Hadoop 2.9.1.
>
>
> Last, thanks Yongjun Zhang, Junping Du, Andrew Wang and Chris Douglas for
> your help and support.
>
>
> Bests,
> Sammi Chen
>
> [1] http://hadoop.apache.org/docs/r2.9.1/index.html
> [2] http://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/
> hadoop-common/release/2.9.1/CHANGES.2.9.1.html
> [3] http://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/
> hadoop-common/release/2.9.1/RELEASENOTES.2.9.1.html
> [4] http://hadoop.apache.org/releases.html#Download
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Thanks Wei-Chiu and Haibo for the feedback!

Good thing is that I have made the following note couple of days ago when I
looked the at branch diff, so we are on the same page:

 496dc57 Revert "YARN-7190. Ensure only NM classpath in 2.x gets TSv2
related hbase jars, not the user classpath. Contributed by Varun Saxena."

*YARN-7190 is not in 3.0.2,  I will include it in 3.0.3 per* the comment
below:
https://issues.apache.org/jira/browse/YARN-7190?focusedCommentId=16457649;
page=com.atlassian.jira.plugin.system.issuetabpanels:
comment-tabpanel#comment-16457649


In addition, I will revert   https://issues.apache.org/
jira/browse/HADOOP-13055 from 3.0.3 since it's a feature.

Best,

--Yongjun

On Tue, May 8, 2018 at 8:57 AM, Haibo Chen <haiboc...@cloudera.com> wrote:

> +1 on adding YARN-7190 to Hadoop 3.0.x despite the fact that it is
> technically incompatible.
> It is critical enough to justify being an exception, IMO.
>
> Added Rohith and Vrushali
>
> On Tue, May 8, 2018 at 6:20 AM, Wei-Chiu Chuang <weic...@apache.org>
> wrote:
>
>> Thanks Yongjun for driving 3.0.3 release!
>>
>> IMHO, could we consider adding YARN-7190
>> <https://issues.apache.org/jira/browse/YARN-7190> into the list?
>> I understand that it is listed as an incompatible change, however, because
>> of this bug, HBase considers the entire Hadoop 3.0.x line not production
>> ready. I feel there's not much point releasing any more 3.0.x releases if
>> downstream projects can't pick it up (after the fact that HBase is one of
>> the most important projects around Hadoop).
>>
>> On Mon, May 7, 2018 at 1:19 PM, Yongjun Zhang <yzh...@cloudera.com>
>> wrote:
>>
>> > Hi Eric,
>> >
>> > Thanks for the feedback, good point. I will try to clean up things, then
>> > cut branch before the release production and vote.
>> >
>> > Best,
>> >
>> > --Yongjun
>> >
>> > On Mon, May 7, 2018 at 8:39 AM, Eric Payne <eric.payne1...@yahoo.com.
>> > invalid
>> > > wrote:
>> >
>> > > >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and
>> vote
>> > > for RC on May 30th
>> > > I much prefer to wait to cut the branch until just before the
>> production
>> > > of the release and the vote. With so many branches, we sometimes miss
>> > > putting critical bug fixes in unreleased branches if the branch is cut
>> > too
>> > > early.
>> > >
>> > > My 2 cents...
>> > > Thanks,
>> > > -Eric Payne
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Monday, May 7, 2018, 12:09:00 AM CDT, Yongjun Zhang <
>> > > yjzhan...@apache.org> wrote:
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Hi All,
>> > >
>> > > >
>> > > We have released Apache Hadoop 3.0.2 in April of this year [1]. Since
>> > then,
>> > > there are quite some commits done to branch-3.0. To further improve
>> the
>> > > quality of release, we plan to do 3.0.3 release now. The focus of
>> 3.0.3
>> > > will be fixing blockers (3), critical bugs (17) and bug fixes (~130),
>> see
>> > > [2].
>> > >
>> > > Usually no new feature should be included for maintenance releases, I
>> > > noticed we have https://issues.apache.org/jira/browse/HADOOP-13055 in
>> > the
>> > > branch classified as new feature. I will talk with the developers to
>> see
>> > if
>> > > we should include it in 3.0.3.
>> > >
>> > > I also noticed that there are more commits in the branch than can be
>> > found
>> > > by query [2], also some commits committed to 3.0.3 do not have their
>> jira
>> > > target release field filled in accordingly. I will go through them to
>> > > update the jira.
>> > >
>> > > >
>> > > We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and vote
>> > for
>> > > RC on May 30th, targeting for Jun 8th release.
>> > >
>> > > >
>> > > Your insights are welcome.
>> > >
>> > > >
>> > > [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07
>> 790.html
>> > >
>> > > > [2] https://issues.apache.org/jira/issues/?filter=12343874  See
>> Note
>> > > below
>> > > Note: seems I need some admin change so that I can make the filter in
>> [2]
>> > > public, I'm working on that. For now, you can use jquery
>> > > (project = hadoop OR project = "Hadoop HDFS" OR project = "Hadoop
>> YARN"
>> > OR
>> > > project = "Hadoop Map/Reduce") AND fixVersion in (3.0.3) ORDER BY
>> > priority
>> > > DESC
>> > >
>> > > Thanks and best regards,
>> > >
>> > > --Yongjun
>> > >
>> > > -
>> > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> > > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> A very happy Hadoop contributor
>>
>
>

Re: Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Hi Eric,

Thanks for the feedback, good point. I will try to clean up things, then
cut branch before the release production and vote.

Best,

--Yongjun

On Mon, May 7, 2018 at 8:39 AM, Eric Payne <eric.payne1...@yahoo.com.invalid
> wrote:

> >  We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and vote
> for RC on May 30th
> I much prefer to wait to cut the branch until just before the production
> of the release and the vote. With so many branches, we sometimes miss
> putting critical bug fixes in unreleased branches if the branch is cut too
> early.
>
> My 2 cents...
> Thanks,
> -Eric Payne
>
>
>
>
>
> On Monday, May 7, 2018, 12:09:00 AM CDT, Yongjun Zhang <
> yjzhan...@apache.org> wrote:
>
>
>
>
>
> Hi All,
>
> >
> We have released Apache Hadoop 3.0.2 in April of this year [1]. Since then,
> there are quite some commits done to branch-3.0. To further improve the
> quality of release, we plan to do 3.0.3 release now. The focus of 3.0.3
> will be fixing blockers (3), critical bugs (17) and bug fixes (~130), see
> [2].
>
> Usually no new feature should be included for maintenance releases, I
> noticed we have https://issues.apache.org/jira/browse/HADOOP-13055 in the
> branch classified as new feature. I will talk with the developers to see if
> we should include it in 3.0.3.
>
> I also noticed that there are more commits in the branch than can be found
> by query [2], also some commits committed to 3.0.3 do not have their jira
> target release field filled in accordingly. I will go through them to
> update the jira.
>
> >
> We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and vote for
> RC on May 30th, targeting for Jun 8th release.
>
> >
> Your insights are welcome.
>
> >
> [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07790.html
>
> > [2] https://issues.apache.org/jira/issues/?filter=12343874  See Note
> below
> Note: seems I need some admin change so that I can make the filter in [2]
> public, I'm working on that. For now, you can use jquery
> (project = hadoop OR project = "Hadoop HDFS" OR project = "Hadoop YARN" OR
> project = "Hadoop Map/Reduce") AND fixVersion in (3.0.3) ORDER BY priority
> DESC
>
> Thanks and best regards,
>
> --Yongjun
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Apache Hadoop 3.0.3 Release plan

[Yongjun Zhang]

Hi All,

>
We have released Apache Hadoop 3.0.2 in April of this year [1]. Since then,
there are quite some commits done to branch-3.0. To further improve the
quality of release, we plan to do 3.0.3 release now. The focus of 3.0.3
will be fixing blockers (3), critical bugs (17) and bug fixes (~130), see
[2].

Usually no new feature should be included for maintenance releases, I
noticed we have https://issues.apache.org/jira/browse/HADOOP-13055 in the
branch classified as new feature. I will talk with the developers to see if
we should include it in 3.0.3.

I also noticed that there are more commits in the branch than can be found
by query [2], also some commits committed to 3.0.3 do not have their jira
target release field filled in accordingly. I will go through them to
update the jira.

>
We plan to cut branch-3.0.3 by the coming Wednesday (May 9th) and vote for
RC on May 30th, targeting for Jun 8th release.

>
Your insights are welcome.

>
[1] https://www.mail-archive.com/general@hadoop.apache.org/msg07790.html

> [2] https://issues.apache.org/jira/issues/?filter=12343874  See Note below
 Note: seems I need some admin change so that I can make the filter in [2]
public, I'm working on that. For now, you can use jquery
(project = hadoop OR project = "Hadoop HDFS" OR project = "Hadoop YARN" OR
project = "Hadoop Map/Reduce") AND fixVersion in (3.0.3) ORDER BY priority
DESC

Thanks and best regards,

--Yongjun

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

[Yongjun Zhang]

Congratulations Andrew and all, a great milestone! Thanks Andrew for
driving it!

--Yongjun

On Tue, Dec 19, 2017 at 9:10 AM, Jonathan Kelly 
wrote:

> Thanks, Andrew!
>
> On Mon, Dec 18, 2017 at 4:54 PM Andrew Wang 
> wrote:
>
> > Thanks for the spot, I just pushed a correct tag. I can't delete the bad
> > tag myself, will ask ASF infra for help.
> >
> > On Mon, Dec 18, 2017 at 4:46 PM, Jonathan Kelly 
> > wrote:
> >
> >> Congrats on the huge release!
> >>
> >> I just noticed, though, that the Github repo does not appear to have the
> >> correct tag for 3.0.0. I see a new tag called "rel/release-" that
> points to
> >> the same commit as "release-3.0.0-RC1"
> >> (c25427ceca461ee979d30edd7a4b0f50718e6533). I assume that should have
> >> actually been called "rel/release-3.0.0" to match the pattern for prior
> >> releases.
> >>
> >> Thanks,
> >> Jonathan Kelly
> >>
> >> On Thu, Dec 14, 2017 at 10:45 AM Andrew Wang 
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I'm pleased to announce that Apache Hadoop 3.0.0 is generally available
> >>> (GA).
> >>>
> >>> 3.0.0 GA consists of 302 bug fixes, improvements, and other
> enhancements
> >>> since 3.0.0-beta1. This release marks a point of quality and stability
> >>> for
> >>> the 3.0.0 release line, and users of earlier 3.0.0-alpha and -beta
> >>> releases
> >>> are encouraged to upgrade.
> >>>
> >>> Looking back, 3.0.0 GA is the culmination of over a year of work on the
> >>> 3.0.0 line, starting with 3.0.0-alpha1 which was released in September
> >>> 2016. Altogether, 3.0.0 incorporates 6,242 changes since 2.7.0.
> >>>
> >>> Users are encouraged to read the overview of major changes
> >>>  in 3.0.0. The GA
> >>> release
> >>> notes
> >>> <
> >>> http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/
> hadoop-common/release/3.0.0/RELEASENOTES.3.0.0.html
> >>> >
> >>>  and changelog
> >>> <
> >>> http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/
> hadoop-common/release/3.0.0/CHANGES.3.0.0.html
> >>> >
> >>> detail
> >>> the changes since 3.0.0-beta1.
> >>>
> >>> The ASF press release provides additional color and highlights some of
> >>> the
> >>> major features:
> >>>
> >>>
> >>> https://globenewswire.com/news-release/2017/12/14/
> 1261879/0/en/The-Apache-Software-Foundation-Announces-
> Apache-Hadoop-v3-0-0-General-Availability.html
> >>>
> >>> Let me end by thanking the many, many contributors who helped with this
> >>> release line. We've only had three major releases in Hadoop's 10 year
> >>> history, and this is our biggest major release ever. It's an incredible
> >>> accomplishment for our community, and I'm proud to have worked with all
> >>> of
> >>> you.
> >>>
> >>> Best,
> >>> Andrew
> >>>
> >>
> >
>

[jira] [Resolved] (HADOOP-14262) rpcTimeOut is not set up correctly in Client thus client doesn't time out

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-14262.

Resolution: Duplicate

> rpcTimeOut is not set up correctly in Client thus client doesn't time out
> -
>
> Key: HADOOP-14262
> URL: https://issues.apache.org/jira/browse/HADOOP-14262
> Project: Hadoop Common
>  Issue Type: Bug
>    Reporter: Yongjun Zhang
>    Assignee: Yongjun Zhang
>
> NameNodeProxies.createNNProxyWithClientProtocol  does
> {code}
>   ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(
> ClientNamenodeProtocolPB.class, version, address, ugi, conf,
> NetUtils.getDefaultSocketFactory(conf),
> org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy,
> fallbackToSimpleAuth).getProxy();
> {code}
> which calls Client.getTimeOut(conf) to get timeout value. 
> Client.getTimeOut(conf) doesn't consider IPC_CLIENT_RPC_TIMEOUT_KEY right 
> now. Thus rpcTimeOut doesn't take effect for relevant RPC calls, and they 
> hang!
> For example, receiveRpcResponse blocked forever at:
> {code}
> Thread 16127: (state = BLOCKED)   
>   
>  - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled 
> frame)   
>  - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 
> (Compiled frame)   
>  - 
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) 
> @bci=5, line=57 (Compiled frame)
>  - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) 
> @bci=35, line=142 (Compiled frame)
>  - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, 
> line=161 (Compiled frame)
>  - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, 
> line=131 (Compiled frame) 
>  - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 
> (Compiled frame)   
>  - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 
> (Compiled frame)   
>  - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, 
> int) @bci=4, line=521 (Compiled frame)
>  - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame) 
>   
>  - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)  
>   
>  - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)
>   
>  - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, 
> line=1081 (Compiled frame) 
>  - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled 
> frame) 
> {code}
> Filing this jira to fix it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14526) Examine code base for cases that exception is thrown from finally block

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-14526:
--

 Summary: Examine code base for cases that exception is thrown from 
finally block
 Key: HADOOP-14526
 URL: https://issues.apache.org/jira/browse/HADOOP-14526
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang


If exception X is thrown in try block, and exception Y is thrown is finally 
block, X will be swallowed. 

In addition, finally block is used to ensure resources are released properly in 
general. If we throw exception from there, some resources may be leaked. So 
it's not recommended to throw exception in the finally block

I caught one today and reported HDFS-11794, creating this jira as a master one 
to catch other similar cases.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14496) Logs for KMS delegation token lifecycle

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-14496.

Resolution: Duplicate

> Logs for KMS delegation token lifecycle
> ---
>
> Key: HADOOP-14496
> URL: https://issues.apache.org/jira/browse/HADOOP-14496
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Yongjun Zhang
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14496) Logs for KMS delegation token lifecycle

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-14496:
--

 Summary: Logs for KMS delegation token lifecycle
 Key: HADOOP-14496
 URL: https://issues.apache.org/jira/browse/HADOOP-14496
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Yongjun Zhang






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14407) DistCp - Introduce a configurable copy buffer size

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-14407.

Resolution: Fixed

> DistCp - Introduce a configurable copy buffer size
> --
>
> Key: HADOOP-14407
> URL: https://issues.apache.org/jira/browse/HADOOP-14407
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Omkar Aradhya K S
>Assignee: Omkar Aradhya K S
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14407.001.patch, HADOOP-14407.002.patch, 
> HADOOP-14407.002.patch, HADOOP-14407.003.patch, 
> HADOOP-14407.004.branch2.patch, HADOOP-14407.004.patch, 
> HADOOP-14407.004.patch, HADOOP-14407.branch2.002.patch, 
> TotalTime-vs-CopyBufferSize.jpg
>
>
> Currently, the RetriableFileCopyCommand has a fixed copy buffer size of just 
> 8KB. We have noticed in our performance tests that with bigger buffer sizes 
> we saw upto ~3x performance boost. Hence, making the copy buffer size a 
> configurable setting via the new parameter .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Jenkins test failure

[Yongjun Zhang]

Thanks a lot Akira! Good catch!

--Yongjun

On Wed, May 17, 2017 at 3:46 PM, Akira Ajisaka <aajis...@apache.org> wrote:

> Hi Yongjun,
>
> Jenkins selects the latest attachment for precommit job regardless of the
> type of the attachment.
>
> The workaround is to attach the patch again.
>
> Regards,
> Akira
>
> On 2017/05/17 18:38, Yongjun Zhang wrote:
>
>> Hi,
>>
>> I saw quite a few jenkins test failure for patches uploaded to jira,
>>
>> For example,
>> https://builds.apache.org/job/PreCommit-HADOOP-Build/12347/console
>>
>> apache-yetus-2971eff/yetus-project/pom.xml
>> Modes:  Sentinel  MultiJDK  Jenkins  Robot  Docker  ResetRepo  UnitTests
>> Processing: HADOOP-14407
>> HADOOP-14407 patch is being downloaded at Wed May 17 19:38:33 UTC 2017
>> from
>>   https://issues.apache.org/jira/secure/attachment/12868556/
>> TotalTime-vs-CopyBufferSize.jpg
>> -> Downloaded
>> ERROR: Unsure how to process HADOOP-14407.
>>
>>
>> Wonder if anyone can help?
>>
>>
>> Thanks.
>>
>>
>> --Yongjun
>>
>>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Jenkins test failure

[Yongjun Zhang]

Hi,

I saw quite a few jenkins test failure for patches uploaded to jira,

For example,
https://builds.apache.org/job/PreCommit-HADOOP-Build/12347/console

apache-yetus-2971eff/yetus-project/pom.xml
Modes:  Sentinel  MultiJDK  Jenkins  Robot  Docker  ResetRepo  UnitTests
Processing: HADOOP-14407
HADOOP-14407 patch is being downloaded at Wed May 17 19:38:33 UTC 2017 from
  
https://issues.apache.org/jira/secure/attachment/12868556/TotalTime-vs-CopyBufferSize.jpg
-> Downloaded
ERROR: Unsure how to process HADOOP-14407.


Wonder if anyone can help?


Thanks.


--Yongjun

[jira] [Reopened] (HADOOP-14407) DistCp - Introduce a configurable copy buffer size

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang reopened HADOOP-14407:


> DistCp - Introduce a configurable copy buffer size
> --
>
> Key: HADOOP-14407
> URL: https://issues.apache.org/jira/browse/HADOOP-14407
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Omkar Aradhya K S
>Assignee: Omkar Aradhya K S
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14407.001.patch
>
>
> Currently, the RetriableFileCopyCommand has a fixed copy buffer size of just 
> 8KB. We have noticed in our performance tests that with bigger buffer sizes 
> we saw upto ~3x performance boost. Hence, making the copy buffer size a 
> configurable setting via the new parameter .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14407) DistCp - Introduce a configurable copy buffer size

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-14407.

Resolution: Information Provided

> DistCp - Introduce a configurable copy buffer size
> --
>
> Key: HADOOP-14407
> URL: https://issues.apache.org/jira/browse/HADOOP-14407
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: tools/distcp
>Affects Versions: 2.9.0
>Reporter: Omkar Aradhya K S
>Assignee: Omkar Aradhya K S
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-14407.001.patch
>
>
> Currently, the RetriableFileCopyCommand has a fixed copy buffer size of just 
> 8KB. We have noticed in our performance tests that with bigger buffer sizes 
> we saw upto ~3x performance boost. Hence, making the copy buffer size a 
> configurable setting via the new parameter .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14333) HADOOP-14104 changed DFSClient API isHDFSEncryptionEnabled, impacted hacky hive code

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-14333:
--

 Summary: HADOOP-14104 changed DFSClient API 
isHDFSEncryptionEnabled, impacted hacky hive code 
 Key: HADOOP-14333
 URL: https://issues.apache.org/jira/browse/HADOOP-14333
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang


Though Hive should be fixed not to access DFSClient which is private to HADOOP, 
removing the throws added by HADOOP-14104 is a quicker solution to unblock hive.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14322) Incorrect host info may be reported in failover message

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-14322:
--

 Summary: Incorrect host info may be reported in failover message
 Key: HADOOP-14322
 URL: https://issues.apache.org/jira/browse/HADOOP-14322
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Reporter: Yongjun Zhang


This may apply to other components, but using HDFS as an example.

When multiple threads use the same DFSClient to make RPC calls, they may report 
incorrect NN host name in the failover message:
{code}
INFO [pool-3-thread-13] retry.RetryInvocationHandler 
(RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete of 
class ClientNamenodeProtocolTranslatorPB over *a.b.c.d*:8020. Trying to fail 
over immediately.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): 
Operation category WRITE is not supported in state standby. Visit 
https://s.apache.org/sbnn-error
{code}

where *a.b.c.d* is the RPC proxy corresponds to the active NN, which confuses 
user to think failover is not behaving correctly. Because *a.b.c.d*  is 
expected to be the proxy corresponding to the standby NN here instead.

The reason is that when the ProxyDescriptor data field of 
RetryInvocationHandler may be shared by multiple threads that do the RPC calls, 
the failover done by one thread (which changed the rpc proxy) may be visible to 
other threads when the other threads report the above message. 

An example sequence: 
# multiple threads start with the same SNN to do RPC calls, 
# all threads discover that a failover is needed, 
# thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
# other threads reports the above message with the proxyInfo changed by thread 
X, and reported ANN instead of SNN in the message.

Some details:

RetryInvocationHandler does the following when failing over:
{code}
  synchronized void failover(long expectedFailoverCount, Method method,
   int callId) {
  // Make sure that concurrent failed invocations only cause a single
  // actual failover.
  if (failoverCount == expectedFailoverCount) {
fpp.performFailover(proxyInfo.proxy);
failoverCount++;
  } else {
LOG.warn("A failover has occurred since the start of call #" + callId
+ " " + proxyInfo.getString(method.getName()));
  }
  proxyInfo = fpp.getProxy();
}
{code}
and changed the proxyInfo in the ProxyDescriptor.

While the log method below report message with ProxyDescriotor's proxyinfo:
{code}
private void log(final Method method, final boolean isFailover,
  final int failovers, final long delay, final Exception ex) {
..
   final StringBuilder b = new StringBuilder()
.append(ex + ", while invoking ")
.append(proxyDescriptor.getProxyInfo().getString(method.getName()));
if (failovers > 0) {
  b.append(" after ").append(failovers).append(" failover attempts");
}
b.append(isFailover? ". Trying to failover ": ". Retrying ");
b.append(delay > 0? "after sleeping for " + delay + "ms.": "immediately.");
{code}
and so does  {{handleException}} method do
{code}
if (LOG.isDebugEnabled()) {
  LOG.debug("Exception while invoking call #" + callId + " "
  + proxyDescriptor.getProxyInfo().getString(method.getName())
  + ". Not retrying because " + retryInfo.action.reason, e);
}
{code}

FailoverProxyProvider
{code}
   public String getString(String methodName) {
  return proxy.getClass().getSimpleName() + "." + methodName
  + " over " + proxyInfo;
}

@Override
public String toString() {
  return proxy.getClass().getSimpleName() + " over " + proxyInfo;
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14198) Should have a way to let PingInputStream to abort

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-14198.

Resolution: Duplicate

> Should have a way to let PingInputStream to abort
> -
>
> Key: HADOOP-14198
> URL: https://issues.apache.org/jira/browse/HADOOP-14198
> Project: Hadoop Common
>  Issue Type: Bug
>    Reporter: Yongjun Zhang
>
> We observed a case that RPC call get stuck, since PingInputStream does the 
> following
> {code}
>  /** This class sends a ping to the remote side when timeout on
>  * reading. If no failure is detected, it retries until at least
>  * a byte is read.
>  */
> private class PingInputStream extends FilterInputStream {
> {code}
> It seems that in this case no data is ever received, and it keeps pinging.
> Should we ping forever here? Maybe we should introduce a config to stop the 
> ping after pinging for certain number of times, and report back timeout, let 
> the caller to retry the RPC?
> Wonder if there is chance the RPC get dropped somehow by the server so no 
> response is ever received.
> See 
> {code}
> Thread 16127: (state = BLOCKED)   
>   
>  - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled 
> frame)   
>  - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 
> (Compiled frame)   
>  - 
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) 
> @bci=5, line=57 (Compiled frame)
>  - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) 
> @bci=35, line=142 (Compiled frame)
>  - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, 
> line=161 (Compiled frame)
>  - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, 
> line=131 (Compiled frame) 
>  - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 
> (Compiled frame)   
>  - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 
> (Compiled frame)   
>  - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, 
> int) @bci=4, line=521 (Compiled frame)
>  - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame) 
>   
>  - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)  
>   
>  - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)
>   
>  - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, 
> line=1081 (Compiled frame) 
>  - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled 
> frame) 
> {code}
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14262) rpcTimeOut is not set up correctly in Client thus client doesn't time out

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-14262:
--

 Summary: rpcTimeOut is not set up correctly in Client thus client 
doesn't time out
 Key: HADOOP-14262
 URL: https://issues.apache.org/jira/browse/HADOOP-14262
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang


NameNodeProxies.createNNProxyWithClientProtocol  does

{code}
  ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(
ClientNamenodeProtocolPB.class, version, address, ugi, conf,
NetUtils.getDefaultSocketFactory(conf),
org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy,
fallbackToSimpleAuth).getProxy();
{code}
which calls Client.getTimeOut(conf) to get timeout value. 

Client.getTimeOut(conf) doesn't consider IPC_CLIENT_RPC_TIMEOUT_KEY right now. 
Thus rpcTimeOut doesn't take effect for relevant RPC calls, and they hang!

For example, receiveRpcResponse blocked forever at:
{code}
Thread 16127: (state = BLOCKED) 

 - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled 
frame)   
 - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 
(Compiled frame)   
 - 
org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) 
@bci=5, line=57 (Compiled frame)
 - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) 
@bci=35, line=142 (Compiled frame)
 - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, 
line=161 (Compiled frame)
 - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, 
line=131 (Compiled frame) 
 - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled 
frame)   
 - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled 
frame)   
 - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, 
int) @bci=4, line=521 (Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)   

 - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)

 - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)  

 - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, 
line=1081 (Compiled frame) 
 - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled 
frame) 
{code}

Filing this jira to fix it.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14198) Should have a way to let PingInputStream to abort

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-14198:
--

 Summary: Should have a way to let PingInputStream to abort
 Key: HADOOP-14198
 URL: https://issues.apache.org/jira/browse/HADOOP-14198
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang


We observed a case that RPC call get stuck, since PingInputStream does the 
following

{code}
 /** This class sends a ping to the remote side when timeout on
 * reading. If no failure is detected, it retries until at least
 * a byte is read.
 */
private class PingInputStream extends FilterInputStream {
{code}

It seems that in this case no data is ever received, and it keeps pinging.

Should we ping forever here? Maybe we should introduce a config to stop the 
ping after pinging for certain number of times, and report back timeout, let 
the caller to retry the RPC?

Wonder if there is chance the RPC get dropped somehow by the server so no 
response is ever received.

See 
{code}
Thread 16127: (state = BLOCKED) 

 - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled 
frame)   
 - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 
(Compiled frame)   
 - 
org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) 
@bci=5, line=57 (Compiled frame)
 - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) 
@bci=35, line=142 (Compiled frame)
 - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, 
line=161 (Compiled frame)
 - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, 
line=131 (Compiled frame) 
 - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled 
frame)   
 - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled 
frame)   
 - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, 
int) @bci=4, line=521 (Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)   

 - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame)

 - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame)  

 - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, 
line=1081 (Compiled frame) 
 - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled 
frame) 
{code}


 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Yongjun Zhang]

Thanks Andrew much for the work here!

+1 (binding).

- Downloaded both binary and src tarballs
- Verified md5 checksum and signature for both
- Built from source tarball
- Deployed 2 pseudo clusters, one with the released tarball and the other
  with what I built from source, and did the following on both:
  - Run basic HDFS operations, snapshots and distcp jobs
  - Run pi job
  - Examined HDFS webui, YARN webui.

 Best,

 --Yongjun


On Tue, Jan 24, 2017 at 3:56 PM, Eric Badger 
wrote:

> +1 (non-binding)
> - Verified signatures and md5- Built from source- Started single-node
> cluster on my mac- Ran some sleep jobs
> Eric
>
> On Tuesday, January 24, 2017 4:32 PM, Yufei Gu 
> wrote:
>
>
>  Hi Andrew,
>
> Thanks for working on this.
>
> +1 (Non-Binding)
>
> 1. Downloaded the binary and verified the md5.
> 2. Deployed it on 3 node cluster with 1 ResourceManager and 2 NodeManager.
> 3. Set YARN to use Fair Scheduler.
> 4. Ran MapReduce jobs Pi
> 5. Verified Hadoop version command output is correct.
>
> Best,
>
> Yufei
>
> On Tue, Jan 24, 2017 at 3:02 AM, Marton Elek 
> wrote:
>
> > ]>
> > > minicluster is kind of weird on filesystems that don't support mixed
> > case, like OS X's default HFS+.
> > >
> > > $  jar tf hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar | grep
> -i
> > license
> > > LICENSE.txt
> > > license/
> > > license/LICENSE
> > > license/LICENSE.dom-documentation.txt
> > > license/LICENSE.dom-software.txt
> > > license/LICENSE.sax.txt
> > > license/NOTICE
> > > license/README.dom.txt
> > > license/README.sax.txt
> > > LICENSE
> > > Grizzly_THIRDPARTYLICENSEREADME.txt
> >
> >
> > I added a patch to https://issues.apache.org/jira/browse/HADOOP-14018 to
> > add the missing META-INF/LICENSE.txt to the shaded files.
> >
> > Question: what should be done with the other LICENSE files in the
> > minicluster. Can we just exclude them (from legal point of view)?
> >
> > Regards,
> > Marton
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>
>
>

[jira] [Created] (HADOOP-13720) Add more info to "token ... is expired" message

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-13720:
--

 Summary: Add more info to "token ... is expired" message
 Key: HADOOP-13720
 URL: https://issues.apache.org/jira/browse/HADOOP-13720
 Project: Hadoop Common
  Issue Type: Bug
  Components: common, security
Reporter: Yongjun Zhang


Currently AbstractDelegationTokenSecretM anager$checkToken does

{code}
  protected DelegationTokenInformation checkToken(TokenIdent identifier)
  throws InvalidToken {
assert Thread.holdsLock(this);
DelegationTokenInformation info = getTokenInfo(identifier);
if (info == null) {
  throw new InvalidToken("token (" + identifier.toString()
  + ") can't be found in cache");
}
if (info.getRenewDate() < Time.now()) {
  throw new InvalidToken("token (" + identifier.toString() + ") is 
expired");
}
return info;
  } 
{code}

When a token is expried, we throw the above exception without printing out the 
{{info.getRenewDate()}} in the message. If we print it out, we could know for 
how long the token has not been renewed. This will help us investigate certain 
issues.

Create this jira as a request to add that part.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Yongjun Zhang]

Hi Sangjin,

Thanks a lot for your work here.

My +1 (binding).

- Downloaded both binary and src tarballs
- Verified md5 checksum and signature for both
- Build from source tarball
- Deployed 2 pseudo clusters, one with the released tarball and the other
with what I built from source, and did the following on both:
- Run basic HDFS operations, and distcp jobs
- Run pi job
- Examined HDFS webui, YARN webui.

Best,

--Yongjun

> > > * verified basic HDFS operations and Pi job.
> > > * Did a sanity check for RM and NM UI.


On Fri, Oct 7, 2016 at 5:08 PM, Sangjin Lee  wrote:

> I'm casting my vote: +1 (binding)
>
> Regards,
> Sangjin
>
> On Fri, Oct 7, 2016 at 3:12 PM, Andrew Wang 
> wrote:
>
> > Thanks to Chris and Sangjin for working on this release.
> >
> > +1 binding
> >
> > * Verified signatures
> > * Built from source tarball
> > * Started HDFS and did some basic ops
> >
> > Thanks,
> > Andrew
> >
> > On Fri, Oct 7, 2016 at 2:50 PM, Wangda Tan  wrote:
> >
> > > Thanks Sangjin for cutting this release!
> > >
> > > +1 (Binding)
> > >
> > > - Downloaded binary tar ball and setup a single node cluster.
> > > - Submit a few applications and which can successfully run.
> > >
> > > Thanks,
> > > Wangda
> > >
> > >
> > > On Fri, Oct 7, 2016 at 10:33 AM, Zhihai Xu 
> > wrote:
> > >
> > > > Thanks Sangjin for creating release 2.6.5 RC1.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > * Downloaded and built from source
> > > > * Verified md5 checksums and signature
> > > > * Deployed a pseudo cluster
> > > > * verified basic HDFS operations and Pi job.
> > > > * Did a sanity check for RM and NM UI.
> > > >
> > > > Thanks
> > > > zhihai
> > > >
> > > > On Fri, Oct 7, 2016 at 8:16 AM, Sangjin Lee 
> wrote:
> > > >
> > > > > Thanks Masatake!
> > > > >
> > > > > Today's the last day for this vote, and I'd like to ask you to try
> > out
> > > > the
> > > > > RC and vote on this today. So far there has been no binding vote.
> > > Thanks
> > > > > again.
> > > > >
> > > > > Regards,
> > > > > Sangjin
> > > > >
> > > > > On Fri, Oct 7, 2016 at 6:45 AM, Masatake Iwasaki <
> > > > > iwasak...@oss.nttdata.co.jp> wrote:
> > > > >
> > > > > > +1(non-binding)
> > > > > >
> > > > > > * verified signature and md5.
> > > > > > * built with -Pnative on CentOS6 and OpenJDK7.
> > > > > > * built documentation and skimmed the contents.
> > > > > > * built rpms by bigtop and ran smoke-tests of hdfs, yarn and
> > > mapreduce
> > > > on
> > > > > > 3-node cluster.
> > > > > >
> > > > > > Thanks,
> > > > > > Masatake Iwasaki
> > > > > >
> > > > > > On 10/3/16 09:12, Sangjin Lee wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> I have pushed a new release candidate (R1) for the Apache Hadoop
> > > 2.6.5
> > > > > >> release (the next maintenance release in the 2.6.x release
> line).
> > > RC1
> > > > > >> contains fixes to CHANGES.txt, and is otherwise identical to
> RC0.
> > > > > >>
> > > > > >> Below are the details of this release candidate:
> > > > > >>
> > > > > >> The RC is available for validation at:
> > > > > >> http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.
> > > > > >>
> > > > > >> The RC tag in git is release-2.6.5-RC1 and its git commit is
> > > > > >> e8c9fe0b4c252caf2ebf1464220599650f119997.
> > > > > >>
> > > > > >> The maven artifacts are staged via repository.apache.org at:
> > > > > >> https://repository.apache.org/content/repositories/
> > > > > orgapachehadoop-1050/.
> > > > > >>
> > > > > >> You can find my public key at
> > > > > >> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
> > > > > >>
> > > > > >> Please try the release and vote. The vote will run for the
> usual 5
> > > > > days. I
> > > > > >> would greatly appreciate your timely vote. Thanks!
> > > > > >>
> > > > > >> Regards,
> > > > > >> Sangjin
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Looking to a Hadoop 3 release

[Yongjun Zhang]

Thanks Andrew for initiating the effort!

+1 on pushing 3.x with extended alpha cycle, and continuing the more stable
2.x releases.

--Yongjun

On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang 
wrote:

> Hi Kai,
>
> Sure, I'm open to it. It's a new major release, so we're allowed to make
> these kinds of big changes. The idea behind the extended alpha cycle is
> that downstreams can give us feedback. This way if we do anything too
> radical, we can address it in the next alpha and have downstreams re-test.
>
> Best,
> Andrew
>
> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai  wrote:
>
> > Thanks Andrew for driving this. Wonder if it's a good chance for
> > HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note
> it's
> > not an incompatible change, but feel better to be done in the major
> release.
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Andrew Wang [mailto:andrew.w...@cloudera.com]
> > Sent: Friday, February 19, 2016 7:04 AM
> > To: hdfs-...@hadoop.apache.org; Kihwal Lee 
> > Cc: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
> > yarn-...@hadoop.apache.org
> > Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi Kihwal,
> >
> > I think there's still value in continuing the 2.x releases. 3.x comes
> with
> > the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> > be beta or GA for some number of months. In the meanwhile, it'd be good
> to
> > keep putting out regular, stable 2.x releases.
> >
> > Best,
> > Andrew
> >
> >
> > On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee  >
> > wrote:
> >
> > > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > > motivations, are we getting rid of branch-2.8?
> > >
> > > Kihwal
> > >
> > >   From: Andrew Wang 
> > >  To: "common-dev@hadoop.apache.org" 
> > > Cc: "yarn-...@hadoop.apache.org" ; "
> > > mapreduce-...@hadoop.apache.org" ;
> > > hdfs-dev 
> > >  Sent: Thursday, February 18, 2016 4:35 PM
> > >  Subject: Re: Looking to a Hadoop 3 release
> > >
> > > Hi all,
> > >
> > > Reviving this thread. I've seen renewed interest in a trunk release
> > > since HDFS erasure coding has not yet made it to branch-2. Along with
> > > JDK8, the shell script rewrite, and many other improvements, I think
> > > it's time to revisit Hadoop 3.0 release plans.
> > >
> > > My overall plan is still the same as in my original email: a series of
> > > regular alpha releases leading up to beta and GA. Alpha releases make
> > > it easier for downstreams to integrate with our code, and making them
> > > regular means features can be included when they are ready.
> > >
> > > I know there are some incompatible changes waiting in the wings (i.e.
> > > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > > If you have changes like this, please set the target version to 3.0.0
> > > and mark them "Incompatible". We can use this JIRA query to track:
> > >
> > >
> > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> > >
> > > There's some release-related stuff that needs to be sorted out
> > > (namely, the new CHANGES.txt and release note generation from Yetus),
> > > but I'd tentatively like to roll the first alpha a month out, so third
> > > week of March.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata 
> > wrote:
> > >
> > > > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > > > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > > > source version to JDK8.
> > > >
> > > > Also, note that releasing from trunk is a way of achieving #3, it's
> > > > not a way of abandoning it.
> > > >
> > > >
> > > >
> > > > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
> > > > 
> > > > wrote:
> > > > > Hi Raymie,
> > > > >
> > > > > Konst proposed just releasing off of trunk rather than cutting a
> > > > branch-2,
> > > > > and there was general agreement there. So, consider #3 abandoned.
> > > > > 1&2
> > > can
> > > > > be achieved at the same time, we just need to avoid using JDK8
> > > > > language features in trunk so things can be backported.
> > > > >
> > > > > Best,
> > > > > Andrew
> > > > >
> > > > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
> > > > > 
> > > > wrote:
> > > > >
> > > > >> In this (and the related threads), I see the following three
> > > > requirements:
> > > > >>
> > > 

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Yongjun Zhang]

Thanks Junping and Allen.

It'd be nice to have HDFS-9629 but I'm ok with option 2, given the fact
that the issue is not critical (and will be addressed in all future
releases), and the concern Allen raised.

Best,

--Yongjun

On Wed, Feb 10, 2016 at 8:37 AM, Allen Wittenauer  wrote:

>
> > On Feb 9, 2016, at 6:27 PM, Junping Du  wrote:
> >
> > Thanks Yongjun for identifying and proposing this change to 2.6.4. I
> think this is the right thing to do and check for following releases. For
> 2.6.4, it seems unnecessary to create another release candidate for this
> issue as we only kicking off a new RC build when last RC has serious
> problem in functionality. The vote progress is quite smoothly so far, so it
> seems unlikely that we will create a new RC. However, I think there are
> still two options here:
> > Option 1:  in final build, adopt change of HDFS-9629 that only updates
> the footer of Web UI to show year 2016.
> > Option 2: skip HDFS-9629 for 2.6.4 and adopt it later for 2.6.5.
> > I prefer Option 1 as this is a very low risky change without affecting
> any functionality, and we allow non-functional changes (like release date,
> etc.) happen on final build after RC passed. I would like to hear the
> voices in community here before acting for the next step. Thoughts?
> >
>
> I’d think having PMC votes apply to what is not actually the final
> artifact is against the ASF rules.
>
>
>

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Yongjun Zhang]

Thanks Junping again for working on this release.

+1 (binding),

- Downloaded source tarball and binary tarball
- Verified signature and checksum for both source and binary tarballs
- Compiled and built a single node cluster
- Run HDFS shell commands to create files
- Run distcp job between this new cluster and some other cluster with older
release successfully.

BTW, in case other people find any issue or request additional important
fixes to include, such that we need new RC, I'd  suggest to to
include HDFS-9629 together to update to release year of Web UI footer
(currently it's 2014).

Thanks.

--Yongjun

On Sat, Feb 6, 2016 at 11:27 PM, Akira AJISAKA 
wrote:

> +1 (binding)
>
> - Downloaded source tarball and binary tarball
> - Verified signatures and checksums
> - Compiled and built a single node cluster
> - Compiled Hive 1.2.1 and Tez 0.7.0/0.8.2 using Hadoop 2.6.4 pom
> successfully
> - Ran some Hive on Tez/MRv2 queries successfully
>
> Thanks,
> Akira
>
> On 2/4/16 11:30, Ted Yu wrote:
>
>> I modified hbase pom.xml (0.98 branch) to point to staged maven artifacts.
>>
>> All unit tests passed.
>>
>> Cheers
>>
>> On Tue, Feb 2, 2016 at 11:01 PM, Junping Du  wrote:
>>
>> Hi community folks,
>>> I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the
>>> next
>>> maintenance release to follow up 2.6.3.) according to email thread of
>>> release plan 2.6.4 [1]. Below is details of this release candidate:
>>>
>>> The RC is available for validation at:
>>> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
>>> *
>>>
>>> The RC tag in git is: release-2.6.4-RC0
>>>
>>> The maven artifacts are staged via repository.apache.org at:
>>> *
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1028/
>>> ?
>>> <
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1028/
>>>
 *

>>>
>>> You can find my public key at:
>>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>>
>>> Please try the release and vote. The vote will run for the usual 5 days.
>>>
>>> Thanks!
>>>
>>>
>>> Cheers,
>>>
>>> Junping
>>>
>>>
>>> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>>>
>>>
>>>
>>
>

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Yongjun Zhang]

Hi Junping,

Thanks a lot for leading the effort on 2.6.4.

We were discussing in HDFS-9629, and have agreement on adding an additional
step in the release process to update the year in the webui footer. See

https://issues.apache.org/jira/browse/HDFS-9629?focusedCommentId=15130043=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15130043

I noticed that the current RC for 2.6.4 still has the year of 2014 (the
example steps in the above link is validated on 2.6.4 RC0). So the steps I
proposed in the link could be applied when we create new RCs.

Thanks Akira for +1 on this proposed additional release step. Once we have
an agreement on the details, we will update the wiki page accordingly.

Best,

--Yongjun


On Tue, Feb 2, 2016 at 11:01 PM, Junping Du  wrote:

> Hi community folks,
>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> *
>
> The RC tag in git is: release-2.6.4-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>  >*
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Yongjun Zhang]

Thanks Vinod and Akira..

I assume we will have at least a committer review of any force-push. Say,
if the commit history already has x, y, z, now we need to force push with
x', we'd better not torget y and z, and if there is conflict putting in y
and z after x', the conflict is better reviewed.

Maybe we should have some guideline documented at a wiki page.

Hope that makes sense.

Best,

--Yongjun


On Thu, Jan 14, 2016 at 6:28 PM, Akira AJISAKA 
wrote:

> Thanks Vinod for the information.
>
> +1 for removing origin/bracnh-2.8 and origin/sjlee/hdfs-merge.
>
> In addition, I'm thinking we need to delete origin/master,
> which was pushed wrongly. I've already deleted
> origin/ajisakaa/common-merge before force-push protection starts.
>
> > — There is a general recommendation from INFRA team to take all of our
> existing release tags under "tags" and copy them to “rel”.
>
> +1 for following the recommendation.
>
> Regards,
> Akira
>
> On 1/15/16 06:26, Vinod Kumar Vavilapalli wrote:
>
>> Hi all,
>>
>> As some of you have noticed, we have an update from ASF infra on git
>> branching policy: We no longer have a ASF wide mandate on disallowing
>> force-pushes on all branches / tags.
>>
>> Summarizing information from the INFRA email for the sake of clarity in
>> the midst of recent confusion
>>   - We now can do force pushes, and branch/tag deletion on any branch or
>> tag except refs/tags/rel
>>   - Any force pushes will be annotated in the commit-email as “[Forced
>> Update!]” for the community to watch out for undesired force-pushes
>>   - Only tags under refs/tags/rel are protected from force-push for the
>> sake of release-provenance: Essentially, the releases that community votes
>> on are archived in their entirety with the development history and we
>> cannot alter that once a tag is created. As one might expect.
>>
>> What this means for us
>>   - Stale branches: There are a few stale branches that got accumulated.
>>  — During this branch moratorium, origin/bracnh-2.8 got created (May
>> be as part of HDFS-8785, can’t say for sure)
>>  — A couple of stale branches that helped 2.6.1 release:
>> origin/sjlee/hdfs-merge and origin/ajisakaa/common-merge
>>  — I’ll wait till EOD tomorrow for any yays/nays and delete them
>>   - Feature branch updates: Developers can now go rebase and force-push
>> their feature branches.
>>   - Mainline branches: Mainline branches like trunk, branch-2 have always
>> been configured to avoid force-pushes. In general, force-push continues to
>> be recommended mainly for feature branches and definitely not on any
>> mainline branches from which we make releases.
>>   - Release tags:
>>  — To follow ASF provenance policy, we will now push the final
>> release tags under refs/tags/rel. We will first push the RC tags under
>> where they reside now (refs/tags) and if the vote passes, the final tag
>> will be created under refs/tags/rel.
>>  — I’ll update our release wiki page
>> http://wiki.apache.org/hadoop/HowToRelease <
>> http://wiki.apache.org/hadoop/HowToRelease> with the same details once I
>> can get 2.7.2 release done using this updated process.
>>   - Existing release tags:
>>  — There is a general recommendation from INFRA team to take all of
>> our existing release tags under "tags" and copy them to “rel”.
>>  — I’ll wait till EOD tomorrow for any yays/nays and copy existing
>> releases under refs/tags/rel following general recommendations.
>>
>> Any comments / thoughts / questions welcome.
>>
>> Thanks
>> +Vinod
>>
>>
>

Re: [UPDATE] New ASF git policy on force-pushes / Tags / Stale branches

[Yongjun Zhang]

Thanks for clarifying Vinod.That sounds good to me.

--Yongjun

On Thu, Jan 14, 2016 at 9:04 PM, Vinod Kumar Vavilapalli <vino...@apache.org
> wrote:

> It kind of got lost through my email, but when I talked about force-push,
> it is almost completely in the context of feature-branch rebase.
>
> We almost never do force-push outside of this context, neither do we
> encourage it. And like I pointed, the fact that the mainline branches
> (trunk, branch-2 etc) are already protected against force-pushes, and the
> latest infra’s change to notify force-pushes as explicitly marked so, are
> good enough for handling them IMO.
>
> Thanks
> +Vinod
>
> > On Jan 14, 2016, at 8:07 PM, Yongjun Zhang <yzh...@cloudera.com> wrote:
> >
> > I assume we will have at least a committer review of any force-push. Say,
> > if the commit history already has x, y, z, now we need to force push with
> > x', we'd better not torget y and z, and if there is conflict putting in y
> > and z after x', the conflict is better reviewed.
>
>

[jira] [Created] (HADOOP-12604) Exception may be swallowed in KMSClientProvider

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-12604:
--

 Summary: Exception may be swallowed in KMSClientProvider
 Key: HADOOP-12604
 URL: https://issues.apache.org/jira/browse/HADOOP-12604
 Project: Hadoop Common
  Issue Type: Bug
  Components: kms
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang


In KMSClientProvider# createConnection
{code}
  try {
is = conn.getInputStream();
ret = mapper.readValue(is, klass);
  } catch (IOException ex) {
if (is != null) {
  is.close(); <== close may throw exception
}
throw ex;
  } finally {
if (is != null) {
  is.close();
}
  }
}
{code}

{{ex}} may be swallowed when {{close}} highlighted in the code throws 
exception.  Thanks [~qwertymaniac] for pointing this out.

BTW, I think we should be able to consolidate the two {{is.close()}} in the 
above code, so we don't close the same stream twice. The one in the {{finally 
block}} may be called after an exception is thrown or not, and it may throw 
exception too, we need to be careful not to swallow exception here too.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12517) Findbugs reported 0 issues, but summary

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-12517:
--

 Summary: Findbugs reported 0 issues, but summary 
 Key: HADOOP-12517
 URL: https://issues.apache.org/jira/browse/HADOOP-12517
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Yongjun Zhang


https://issues.apache.org/jira/browse/HDFS-9231?focusedCommentId=14975559=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14975559

stated -1 for findbugs, however, 

https://builds.apache.org/job/PreCommit-HDFS-Build/13205/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html

says 0.

Thanks a lot for looking into.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12103) Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-12103:
--

 Summary: Small refactoring of DelegationTokenAuthenticationFilter 
to allow code sharing
 Key: HADOOP-12103
 URL: https://issues.apache.org/jira/browse/HADOOP-12103
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.7.1
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Priority: Minor


This is the hadoop-common portion change for HDFS-8337 patch rev 003.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: set up jenkins test for branch-2

[Yongjun Zhang]

Thanks Sean and Allen!

I was not aware of that there is already a way to trigger branch-2 test.
Good to know.

There are multiple solutions here:

1. When posting patches, we can post two versions of patches, one for
trunk, and one for branch-2, so to trigger two pre-commit jenkins test for
both branches. This would help catching issue before committing. However,
it's going to be more workload on the testing machines, so alternatively,
we can probably defer the branch-2 testing until committing to trunk and
before branch-2 testing.

2. Only post patch for trunk, we cherry-pick to branch-2 when committing.
We can set up periodic jenkins test (such as nightly) for branch-2 to catch
problem periodically. But that's basically being ignored by us as Allen
pointed out.

The goal in my mind is to try our best to have both commits to trunk and
branch-2 to pass jenkins test, thus help on quality.

Thanks.

--Yongjun


On Mon, Jun 15, 2015 at 7:21 AM, Allen Wittenauer a...@altiscale.com wrote:


 On Jun 14, 2015, at 1:00 PM, Yongjun Zhang yzh...@cloudera.com wrote:

  From time to time we saw changes that work fine in trunk but not
 branch-2,
  and we don't catch the issue in a timely manner. The difference between
  trunk and branch-2 is sufficient to justify periodic jenkins test and
 even
  pre-commit test for branch-2.

 What is your expected outcome?

 The current nightly builds have been failing pretty consistently
 for a while now.  Adding another one isn't going to make the situation any
 better since everyone pretty much ignores them.  As I said at the HDFS
 BOF:  we desperately need a moratorium on adding new features and need to
 work on stabilizing both branch-2 and trunk.  We seem to be in a
 quantity over quality mode and it's slowly killing the project.  We
 haven't had a stable release since 2.4.1, which is approaching the 1 year
 mark.

 (Yes, I know.  Various vendors have shipped releases since 2.4.1
 as part of their distributions.  Before saying anything, perhaps you should
 spend some time with your support folks….)

 As Sean pointed out, precommit for branch-2 has been working for a
 while now if one provides enough hints to test-patch as part of the patch
 name.

set up jenkins test for branch-2

[Yongjun Zhang]

Hi,

We touched this topic before but it was put on hold. I'd like to bring it
to our attention again.

From time to time we saw changes that work fine in trunk but not branch-2,
and we don't catch the issue in a timely manner. The difference between
trunk and branch-2 is sufficient to justify periodic jenkins test and even
pre-commit test for branch-2.

I created https://issues.apache.org/jira/browse/INFRA-9226 earlier but I'm
not sure who are the right folks to take care of it.

Any one could help follow-up?

Thanks a lot and best regards,

--Yongjun

Re: REMINDER! REGISTRATIONS CLOSING 5/6!

[Yongjun Zhang]

Thanks Allen a lot for organizing the Haoop Bug Bash, very happy to see
many folks by face!

Wish you all a nice weekend!


--Yongjun


On Tue, May 5, 2015 at 9:31 PM, Allen Wittenauer a...@altiscale.com wrote:


 On May 5, 2015, at 8:10 PM, Allen Wittenauer a...@altiscale.com wrote:

* We’ll be closing registrations to the Bug Bash on May
 6th at 3PM Pacific time.  So make sure you do it son:
 https://www.eventbrite.com/e/apache-hadoop-global-bug-bash-tickets-16507188445

 That should be *noon* Pacific time.  So just do it already, ok?

 [I can’t tell time.  Someone should buy me an Apple Watch Edition
 or something.]

Re: Save the Date: Apache Hadoop Bug Bash / May 8th!

[Yongjun Zhang]

Great idea Allen, big +1!

Thanks.

--Yongjun

On Thu, Mar 19, 2015 at 11:06 AM, Tsuyoshi Ozawa oz...@apache.org wrote:

 Hi Allen,

 Thank you for the great suggestion. Let's do this!

 Thanks,
 - Tsuyoshi

 On Fri, Mar 20, 2015 at 2:34 AM, Allen Wittenauer a...@altiscale.com
 wrote:
 
  Hi folks,
 
  There are ~6,000 Hadoop JIRA issues that have gone unaddressed,
 including ~900 with patches waiting to be reviewed.  Among other things,
 this lack of attention to our backlog is making the Hadoop project very
 unfriendly to contributors--which is ultimately very unhealthy for the
 project.
 
  In hopes of resetting our community habits a bit, a bunch of us
 are organizing a bug bash of sorts on Friday May 8th, the day after HBase
 Con.  We're thinking about holding an in-person event someplace in the San
 Francisco Bay Area... but we definitely want this to be a virtual event for
 participants around the world!
 
  As details form, we'll let you know.  But we wanted to send an
 initial notice so you can save the date, and also share your ideas on how
 we can make this event successful.
 
 
  Thanks!

Re: about CHANGES.txt

[Yongjun Zhang]

Thanks Ravi and Colin for the feedback.

Hi Allen,

You pointed out that git log has problem when dealing with branch that
has merges, would you please elaborate the problem?

Thanks.

--Yongjun

On Mon, Mar 16, 2015 at 7:08 PM, Colin McCabe cmcc...@alumni.cmu.edu
wrote:

 Branch merges made it hard to access change history on subversion
 sometimes.

 You can read the tale of woe here:

 http://programmers.stackexchange.com/questions/206016/maintaining-svn-history-for-a-file-when-merge-is-done-from-the-dev-branch-to-tru

 Excerpt:
 prior to Subversion 1.8. The files in the branch and the files in
 trunk are copies and Subversion keeps track with svn log only for
 specific files, not across branches.

 I think that's how the custom of CHANGES.txt started, and it was
 cargo-culted forward into the git era despite not serving much purpose
 any more these days (in my opinion.)

 best,
 Colin

 On Mon, Mar 16, 2015 at 4:49 PM, Ravi Prakash ravi...@ymail.com wrote:
  +1 for automating the information contained in CHANGES.txt. There are
 some changes which go in without JIRAs sometimes (CVEs eg.) . I like git
 log because its the absolute source of truth (cryptographically secure,
 audited, distributed, yadadada). We could always use git hooks to force a
 commit message format.
  a) cherry-picks have the same message (by default) as the original)b)
 I'm not sure why branch-mergers would be a problem?c) Whoops I missed
 something in the previous commit wouldn't happen if our hooks were
 smartishd) no identification of what type of commit it was without hooking
 into JIRA anyway. This would be in the format of the commit message
 
  Either way I think would be an improvement.
  Thanks for your ideas folks
 
 
 
   On Monday, March 16, 2015 11:51 AM, Colin P. McCabe 
 cmcc...@apache.org wrote:
 
 
   +1 for generating CHANGES.txt from JIRA and/or git as part of making a
  release.  Or just dropping it altogether.  Keeping it under version
  control creates lot of false conflicts whenever submitting a patch and
  generally makes committing minor changes unpleasant.
 
  Colin
 
  On Sat, Mar 14, 2015 at 8:36 PM, Yongjun Zhang yzh...@cloudera.com
 wrote:
  Hi Allen,
 
  Thanks a lot for your input!
 
  Looks like problem a, c, d you listed is not too bad, assuming we can
 solve
  d by pulling this info from jira as Sean pointed out.
 
  Problem b (branch mergers) seems to be a real one, and your approach of
  using JIRA system to build changes.txt is a reasonably good way. This
 does
  count on that we update jira accurately. Since this update is a manual
  process, it's possible to have inconsistency, but may be not too bad.
 Since
  any mistake found here can be remedied by fixing the jira side and
  refreshing the result.
 
  I wonder if we as a community should switch to using your way, and save
  committer's effort of taking care of CHANGES.txt (quite some save IMO).
  Hope more people can share their thoughts.
 
  Thanks.
 
  --Yongjun
 
  On Fri, Mar 13, 2015 at 4:45 PM, Allen Wittenauer a...@altiscale.com
 wrote:
 
 
  I think the general consensus is don’t include the changes.txt file in
  your commit. It won’t be correct for both branches if such a commit is
  destined for both. (No, the two branches aren’t the same.)
 
  No, git log isn’t more accurate.  The problems are:
 
  a) cherry picks
  b) branch mergers
  c) “whoops i missed something in that previous commit”
  d) no identification of what type of commit it was without hooking into
  JIRA anyway.
 
  This is why I prefer building the change log from JIRA.  We already
 build
  release notes from JIRA, BTW.  (Not that anyone appears to read them
 given
  the low quality of our notes…)  Anyway, here’s what I’ve been
  building/using as changes.txt and release notes:
 
  https://github.com/aw-altiscale/hadoop-release-metadata
 
  I try to update these every day. :)
 
  On Mar 13, 2015, at 4:07 PM, Yongjun Zhang yzh...@cloudera.com
 wrote:
 
   Thanks Esteban, I assume this report gets info purely from the jira
   database, but not git log of a branch, right?
  
   I hope we get the info from git log of a release branch because
 that'd
  be
   more accurate.
  
   --Yongjun
  
   On Fri, Mar 13, 2015 at 3:11 PM, Esteban Gutierrez 
 este...@cloudera.com
  
   wrote:
  
   JIRA already provides a report:
  
  
  
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12327179styleName=HtmlprojectId=12310240
  
  
   cheers,
   esteban.
  
  
  
  
   --
   Cloudera, Inc.
  
  
   On Fri, Mar 13, 2015 at 3:01 PM, Sean Busbey bus...@cloudera.com
  wrote:
  
   So long as you include the issue number, you can automate pulling
 the
   type
   from jira directly instead of putting it in the message.
  
   On Fri, Mar 13, 2015 at 4:49 PM, Yongjun Zhang 
 yzh...@cloudera.com
   wrote:
  
   Hi,
  
   I found that changing CHANGES.txt when committing a jira is error
  prone
   because of the different sections in the file, and sometimes we

Re: about CHANGES.txt

[Yongjun Zhang]

Hi Allen,

To make it simpler for you to address my question, does git log miss
commits or report redundant commits on a branch with merged
ancestor branches?

Thanks.

--Yongjun


On Tue, Mar 17, 2015 at 11:21 AM, Allen Wittenauer a...@altiscale.com wrote:


 Nope.  I’m not particularly in the mood to write a book about a
 topic that I’ve beat to death in private conversations over the past 6
 months other than highlighting that any solution needs to be able to work
 against scenarios like we had 3 years ago with four active release branches
 + trunk.

 On Mar 17, 2015, at 10:56 AM, Yongjun Zhang yzh...@cloudera.com wrote:

  Thanks Ravi and Colin for the feedback.
 
  Hi Allen,
 
  You pointed out that git log has problem when dealing with branch that
  has merges, would you please elaborate the problem?
 
  Thanks.
 
  --Yongjun
 
  On Mon, Mar 16, 2015 at 7:08 PM, Colin McCabe cmcc...@alumni.cmu.edu
  wrote:
 
  Branch merges made it hard to access change history on subversion
  sometimes.
 
  You can read the tale of woe here:
 
 
 http://programmers.stackexchange.com/questions/206016/maintaining-svn-history-for-a-file-when-merge-is-done-from-the-dev-branch-to-tru
 
  Excerpt:
  prior to Subversion 1.8. The files in the branch and the files in
  trunk are copies and Subversion keeps track with svn log only for
  specific files, not across branches.
 
  I think that's how the custom of CHANGES.txt started, and it was
  cargo-culted forward into the git era despite not serving much purpose
  any more these days (in my opinion.)
 
  best,
  Colin
 
  On Mon, Mar 16, 2015 at 4:49 PM, Ravi Prakash ravi...@ymail.com
 wrote:
  +1 for automating the information contained in CHANGES.txt. There are
  some changes which go in without JIRAs sometimes (CVEs eg.) . I like git
  log because its the absolute source of truth (cryptographically secure,
  audited, distributed, yadadada). We could always use git hooks to force
 a
  commit message format.
  a) cherry-picks have the same message (by default) as the original)b)
  I'm not sure why branch-mergers would be a problem?c) Whoops I missed
  something in the previous commit wouldn't happen if our hooks were
  smartishd) no identification of what type of commit it was without
 hooking
  into JIRA anyway. This would be in the format of the commit message
 
  Either way I think would be an improvement.
  Thanks for your ideas folks
 
 
 
  On Monday, March 16, 2015 11:51 AM, Colin P. McCabe 
  cmcc...@apache.org wrote:
 
 
  +1 for generating CHANGES.txt from JIRA and/or git as part of making a
  release.  Or just dropping it altogether.  Keeping it under version
  control creates lot of false conflicts whenever submitting a patch and
  generally makes committing minor changes unpleasant.
 
  Colin
 
  On Sat, Mar 14, 2015 at 8:36 PM, Yongjun Zhang yzh...@cloudera.com
  wrote:
  Hi Allen,
 
  Thanks a lot for your input!
 
  Looks like problem a, c, d you listed is not too bad, assuming we can
  solve
  d by pulling this info from jira as Sean pointed out.
 
  Problem b (branch mergers) seems to be a real one, and your approach
 of
  using JIRA system to build changes.txt is a reasonably good way. This
  does
  count on that we update jira accurately. Since this update is a manual
  process, it's possible to have inconsistency, but may be not too bad.
  Since
  any mistake found here can be remedied by fixing the jira side and
  refreshing the result.
 
  I wonder if we as a community should switch to using your way, and
 save
  committer's effort of taking care of CHANGES.txt (quite some save
 IMO).
  Hope more people can share their thoughts.
 
  Thanks.
 
  --Yongjun
 
  On Fri, Mar 13, 2015 at 4:45 PM, Allen Wittenauer a...@altiscale.com
  wrote:
 
 
  I think the general consensus is don’t include the changes.txt file
 in
  your commit. It won’t be correct for both branches if such a commit
 is
  destined for both. (No, the two branches aren’t the same.)
 
  No, git log isn’t more accurate.  The problems are:
 
  a) cherry picks
  b) branch mergers
  c) “whoops i missed something in that previous commit”
  d) no identification of what type of commit it was without hooking
 into
  JIRA anyway.
 
  This is why I prefer building the change log from JIRA.  We already
  build
  release notes from JIRA, BTW.  (Not that anyone appears to read them
  given
  the low quality of our notes…)  Anyway, here’s what I’ve been
  building/using as changes.txt and release notes:
 
  https://github.com/aw-altiscale/hadoop-release-metadata
 
  I try to update these every day. :)
 
  On Mar 13, 2015, at 4:07 PM, Yongjun Zhang yzh...@cloudera.com
  wrote:
 
  Thanks Esteban, I assume this report gets info purely from the jira
  database, but not git log of a branch, right?
 
  I hope we get the info from git log of a release branch because
  that'd
  be
  more accurate.
 
  --Yongjun
 
  On Fri, Mar 13, 2015 at 3:11 PM, Esteban Gutierrez 
  este...@cloudera.com

Re: about CHANGES.txt

[Yongjun Zhang]

Hi Allen,

Thanks a lot for your input!

Looks like problem a, c, d you listed is not too bad, assuming we can solve
d by pulling this info from jira as Sean pointed out.

Problem b (branch mergers) seems to be a real one, and your approach of
using JIRA system to build changes.txt is a reasonably good way. This does
count on that we update jira accurately. Since this update is a manual
process, it's possible to have inconsistency, but may be not too bad. Since
any mistake found here can be remedied by fixing the jira side and
refreshing the result.

I wonder if we as a community should switch to using your way, and save
committer's effort of taking care of CHANGES.txt (quite some save IMO).
Hope more people can share their thoughts.

Thanks.

--Yongjun

On Fri, Mar 13, 2015 at 4:45 PM, Allen Wittenauer a...@altiscale.com wrote:


 I think the general consensus is don’t include the changes.txt file in
 your commit. It won’t be correct for both branches if such a commit is
 destined for both. (No, the two branches aren’t the same.)

 No, git log isn’t more accurate.  The problems are:

 a) cherry picks
 b) branch mergers
 c) “whoops i missed something in that previous commit”
 d) no identification of what type of commit it was without hooking into
 JIRA anyway.

 This is why I prefer building the change log from JIRA.  We already build
 release notes from JIRA, BTW.  (Not that anyone appears to read them given
 the low quality of our notes…)  Anyway, here’s what I’ve been
 building/using as changes.txt and release notes:

 https://github.com/aw-altiscale/hadoop-release-metadata

 I try to update these every day. :)

 On Mar 13, 2015, at 4:07 PM, Yongjun Zhang yzh...@cloudera.com wrote:

  Thanks Esteban, I assume this report gets info purely from the jira
  database, but not git log of a branch, right?
 
  I hope we get the info from git log of a release branch because that'd
 be
  more accurate.
 
  --Yongjun
 
  On Fri, Mar 13, 2015 at 3:11 PM, Esteban Gutierrez este...@cloudera.com
 
  wrote:
 
  JIRA already provides a report:
 
 
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12327179styleName=HtmlprojectId=12310240
 
 
  cheers,
  esteban.
 
 
 
 
  --
  Cloudera, Inc.
 
 
  On Fri, Mar 13, 2015 at 3:01 PM, Sean Busbey bus...@cloudera.com
 wrote:
 
  So long as you include the issue number, you can automate pulling the
  type
  from jira directly instead of putting it in the message.
 
  On Fri, Mar 13, 2015 at 4:49 PM, Yongjun Zhang yzh...@cloudera.com
  wrote:
 
  Hi,
 
  I found that changing CHANGES.txt when committing a jira is error
 prone
  because of the different sections in the file, and sometimes we forget
  about changing this file.
 
  After all, git log would indicate the history of a branch. I wonder if
  we
  could switch to a new method:
 
  1. When committing, ensure the message include the type of the jira,
  New
  Feature, Bug Fixes, Improvement etc.
 
  2. No longer need to make changes to CHANGES.txt for each commit
 
  3. Before releasing a branch, create the CHANGES.txt by using git
 log
  command for the given branch..
 
  Thanks.
 
  --Yongjun
 
 
 
 
  --
  Sean
 
 

Re: about CHANGES.txt

[Yongjun Zhang]

Thanks Sean, that'd be even better!

--Yongjun

On Fri, Mar 13, 2015 at 3:01 PM, Sean Busbey bus...@cloudera.com wrote:

 So long as you include the issue number, you can automate pulling the type
 from jira directly instead of putting it in the message.

 On Fri, Mar 13, 2015 at 4:49 PM, Yongjun Zhang yzh...@cloudera.com
 wrote:

  Hi,
 
  I found that changing CHANGES.txt when committing a jira is error prone
  because of the different sections in the file, and sometimes we forget
  about changing this file.
 
  After all, git log would indicate the history of a branch. I wonder if we
  could switch to a new method:
 
  1. When committing, ensure the message include the type of the jira, New
  Feature, Bug Fixes, Improvement etc.
 
  2. No longer need to make changes to CHANGES.txt for each commit
 
  3. Before releasing a branch, create the CHANGES.txt by using git log
  command for the given branch..
 
  Thanks.
 
  --Yongjun
 



 --
 Sean

Re: about CHANGES.txt

[Yongjun Zhang]

Thanks Esteban, I assume this report gets info purely from the jira
database, but not git log of a branch, right?

I hope we get the info from git log of a release branch because that'd be
more accurate.

--Yongjun

On Fri, Mar 13, 2015 at 3:11 PM, Esteban Gutierrez este...@cloudera.com
wrote:

 JIRA already provides a report:


 https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12327179styleName=HtmlprojectId=12310240


 cheers,
 esteban.




 --
 Cloudera, Inc.


 On Fri, Mar 13, 2015 at 3:01 PM, Sean Busbey bus...@cloudera.com wrote:

  So long as you include the issue number, you can automate pulling the
 type
  from jira directly instead of putting it in the message.
 
  On Fri, Mar 13, 2015 at 4:49 PM, Yongjun Zhang yzh...@cloudera.com
  wrote:
 
   Hi,
  
   I found that changing CHANGES.txt when committing a jira is error prone
   because of the different sections in the file, and sometimes we forget
   about changing this file.
  
   After all, git log would indicate the history of a branch. I wonder if
 we
   could switch to a new method:
  
   1. When committing, ensure the message include the type of the jira,
 New
   Feature, Bug Fixes, Improvement etc.
  
   2. No longer need to make changes to CHANGES.txt for each commit
  
   3. Before releasing a branch, create the CHANGES.txt by using git log
   command for the given branch..
  
   Thanks.
  
   --Yongjun
  
 
 
 
  --
  Sean
 

Re: Looking to a Hadoop 3 release

[Yongjun Zhang]

Thanks all.

There is an open issue HDFS-6962 (ACLs inheritance conflicts with
umaskmode), for which the incompatibility appears to make it not suitable
for 2.x and it's targetted 3.0, please see:

https://issues.apache.org/jira/browse/HDFS-6962?focusedCommentId=14335418page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14335418

Best,

--Yongjun


On Wed, Mar 4, 2015 at 8:13 PM, Allen Wittenauer a...@altiscale.com wrote:


 One of the questions that keeps popping up is “what exactly is in trunk?”

 As some may recall, I had done some experiments creating the change log
 based upon JIRA.  While the interest level appeared to be approaching zero,
 I kept playing with it a bit and eventually also started playing with the
 release notes script (for various reasons I won’t bore you with.)

 In any case, I’ve started posting the results of these runs on one of my
 github repos if anyone was wanting a quick reference as to JIRA’s opinion
 on the matter:

 https://github.com/aw-altiscale/hadoop-release-metadata/tree/master/3.0.0

Re: Looking to a Hadoop 3 release

[Yongjun Zhang]

Thanks Andrew for the proposal.

+1, and I will be happy to help.

--Yongjun




On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:

 Hi devs,

 It's been a year and a half since 2.x went GA, and I think we're about due
 for a 3.x release.
 Notably, there are two incompatible changes I'd like to call out, that will
 have a tremendous positive impact for our users.

 First, classpath isolation being done at HADOOP-11656, which has been a
 long-standing request from many downstreams and Hadoop users.

 Second, bumping the source and target JDK version to JDK8 (related to
 HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two
 months from now). In the past, we've had issues with our dependencies
 discontinuing support for old JDKs, so this will future-proof us.

 Between the two, we'll also have quite an opportunity to clean up and
 upgrade our dependencies, another common user and developer request.

 I'd like to propose that we start rolling a series of monthly-ish series of
 3.0 alpha releases ASAP, with myself volunteering to take on the RM and
 other cat herding responsibilities. There are already quite a few changes
 slated for 3.0 besides the above (for instance the shell script rewrite) so
 there's already value in a 3.0 alpha, and the more time we give downstreams
 to integrate, the better.

 This opens up discussion about inclusion of other changes, but I'm hoping
 to freeze incompatible changes after maybe two alphas, do a beta (with no
 further incompat changes allowed), and then finally a 3.x GA. For those
 keeping track, that means a 3.x GA in about four months.

 I would also like to stress though that this is not intended to be a big
 bang release. For instance, it would be great if we could maintain wire
 compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
 branch-2 and branch-3 similar also makes backports easier, since we're
 likely maintaining 2.x for a while yet.

 Please let me know any comments / concerns related to the above. If people
 are friendly to the idea, I'd like to cut a branch-3 and start working on
 the first alpha.

 Best,
 Andrew

Re: 2.7 status

[Yongjun Zhang]

Hi,

Thanks for working on 2.7 release.

Currently the fallback from KerberosAuthenticator to
PseudoAuthenticator  is enabled by default in a hardcoded way. HAOOP-10895
changes the default and requires applications (such as oozie) to set a
config property or call an API to enable the fallback.

This jira has been reviewed, and almost ready to get in. However, there
is a concern that we have to change the relevant applications. Please see
my comment here:

https://issues.apache.org/jira/browse/HADOOP-10895?focusedCommentId=14321823page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14321823

Any of your comments will be highly appreciated. This jira was postponed
from 2.6. I think it should be no problem to skip 2.7. But your comments
would help us to decide what to do with this jira for future releases.

Thanks.

--Yongjun


On Sun, Mar 1, 2015 at 11:58 AM, Arun Murthy a...@hortonworks.com wrote:

 Sounds good, thanks for the help Vinod!

 Arun

 
 From: Vinod Kumar Vavilapalli
 Sent: Sunday, March 01, 2015 11:43 AM
 To: Hadoop Common; Jason Lowe; Arun Murthy
 Subject: Re: 2.7 status

 Agreed. How about we roll an RC end of this week? As a Java 7+ release
 with features, patches that already got in?

 Here's a filter tracking blocker tickets -
 https://issues.apache.org/jira/issues/?filter=12330598. Nine open now.

 +Arun
 Arun, I'd like to help get 2.7 out without further delay. Do you mind me
 taking over release duties?

 Thanks,
 +Vinod
 
 From: Jason Lowe jl...@yahoo-inc.com.INVALID
 Sent: Friday, February 13, 2015 8:11 AM
 To: common-dev@hadoop.apache.org
 Subject: Re: 2.7 status

 I'd like to see a 2.7 release sooner than later.  It has been almost 3
 months since Hadoop 2.6 was released, and there have already been 634 JIRAs
 committed to 2.7.  That's a lot of changes waiting for an official release.

 https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2Chdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolution%3DFixed
 Jason

   From: Sangjin Lee sj...@apache.org
  To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org
  Sent: Tuesday, February 10, 2015 1:30 PM
  Subject: 2.7 status

 Folks,

 What is the current status of the 2.7 release? I know initially it started
 out as a java-7 only release, but looking at the JIRAs that is very much
 not the case.

 Do we have a certain timeframe for 2.7 or is it time to discuss it?

 Thanks,
 Sangjin

[jira] [Created] (HADOOP-11597) Factor OSType out from Shell: change in common

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-11597:
--

 Summary: Factor OSType out from Shell: change in common
 Key: HADOOP-11597
 URL: https://issues.apache.org/jira/browse/HADOOP-11597
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: util
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11551) Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-11551:
--

 Summary: Let nightly jenkins jobs run the tool of HADOOP-11045 and 
include the result in the job report
 Key: HADOOP-11551
 URL: https://issues.apache.org/jira/browse/HADOOP-11551
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, tools
Reporter: Yongjun Zhang


This jira is to propose running the tool created with HADOOP-11045 at the end 
of jenkins test job - I am thinking about trunk jobs currently - and report the 
results at the job report. This way when we look at test failure, we can tell 
the failure pattern, and whether failed test is likely a flaky test or not.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

supportability JIRAs

[Yongjun Zhang]

Hi,

When creating jira, there is a Labels entry where we could label the jira
as supportability. It would help to do this, so we could possibly give
priority to fix them for better supportability,  especially for those JIRAs
that tend to be small but very high-impact for end users.

The types of jiras that we can label supportability are (see some
detailed description at the bottom of this mail):

- improving logs and error messages
- simplifying workflows for admins
- reducing configuration complexity
- adding new metrics
- ...

For example,

HDFS-7281 https://issues.apache.org/jira/browse/HDFS-7281Missing block is
marked as corrupted block

HDFS-7497 https://issues.apache.org/jira/browse/HDFS-7497Inconsistent
report of decommissioning DataNodes between dfsadmin and NameNode webui

HDFS-6959 https://issues.apache.org/jira/browse/HDFS-6959Make the HDFS
home directory location customizable.

HDFS-6403 https://issues.apache.org/jira/browse/HDFS-6403Add metrics for
log warnings reported by JVM pauses

This email serves as an proposal to do this kind of labeling when creating
new jiras. We could also go back to label old jiras for reference when time
allows.

Comments are welcome.

Thanks.

--Yongjun

Below is a bit more detailed description of some relevant scenarios (thanks
to Andrew Wang and Todd Lipcon):

1) In the presence of configuration errors, detecting them preemptively
before they result in the system getting into a funky state. For example,
we used to have a possible configuration where the NN would start up bound
to 0.0.0.0 and then advertise 0.0.0.0 to the SNN as its remote IP. This
meant that checkpointing wouldn't work, but would fail with confusing
errors. Aborting at startup made this easier to support.

2) In the presence of environmental issues, detecting them and giving
meaningful errors. For example, stuff like the GC Pause monitor that's in
the NN now is helpful because when something goes wrong, you have a smoking
gun. (even though it's not exactly an NN bug that GC happens, in some cases)

3) Changing non-specific error messages to specific ones. For example,
we've had cases before where we throw an NPE, and the fix is to check for
null and throw an IllegalArgumentException with a nice message or
something. It wasn't a bug that the system failed with that particular
config, but the error message tells the user/supporter exactly what to do
to fix it.

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Hi Konst,

Thanks for digging out this info and share!

Interesting to see that the setting is not configurable and  it is not a
simple fix.

Best,

--Yongjun




On Mon, Dec 15, 2014 at 5:43 PM, Konstantin Shvachko shv.had...@gmail.com
wrote:

 Did some research on changing the default order of attachments.
 It is not a configuration or INFRA issue.
 Turned out to be a controversial topic in the Jira itself, which was
 explicitly rejected by the developers. With many users unsatisfied.

 https://jira.atlassian.com/browse/JRA-28290

 I thought it should be a simple thing to fix...
 Oh well. Revision numbers is the way to go then for now.

 Thanks,
 --Konst

 On Mon, Dec 15, 2014 at 11:55 AM, Andrew Wang andrew.w...@cloudera.com
 wrote:
 
  I'm all for changing the default sort order, but it doesn't address the
  point that Steve and I brought up about local downloads.
 
  If you want to push on the INFRA JIRA though, please feel free. I'm +1
 for
  that.
 
  Best,
  Andrew
 
  On Mon, Dec 15, 2014 at 11:40 AM, Konstantin Shvachko 
  shv.had...@gmail.com
  wrote:
  
   Guys,
  
   I agree that revision numbers are useful if you need to reference a
   particular attachment. As well as with all your other arguments.
   My general point is that the infrastructure we use should be convenient
  for
   the users to do such simple things automatically. Rather than us
   introducing rules to overcome certain shortcomings of the tool. I think
  if
   the Attachments list was
   1. ordered by date rather than by name, and
   2. enumerated, like subtasks are
   then it would have solved the issue discussed here.
  
   I did communicate changing the default ordering for attachments with
  INFRA
   some time ago. Don't remember if I created a jira. Should we open one
  now?
  
   Thanks,
   --Konst
  
   On Sun, Dec 14, 2014 at 6:44 AM, Steve Loughran 
 ste...@hortonworks.com
   wrote:
   
a couple more benefits
   
1. when you post a patch you can add a comment like patch 003 killed
  NPE
in auth, and the comment history then integrations with the
 revisions.
   You
can also do this in your private git repository, so correlate commits
   there
with patch versions.
   
2. they list in creation order in a directory.
   
#2 matters for me as when I create patches I stick them in a dir
  specific
to that JIRA; I can work out what the highest number is and increment
  it
   by
one for creating a new one...yet retain the whole patch history
  locally.
   
I also download external patches to review  apply to an incoming/
 dir;
numbering helps me manage that  to verify that I really am applying
  the
relevant patch.
   
Doesn't mean we should change the order though. I don't think that is
something you can do on a per-project basis, so take it to
   infrastructure@
   
   
On 14 December 2014 at 01:33, Yongjun Zhang yzh...@cloudera.com
  wrote:
   
 Hi Konst,

 Thanks for the good suggestion, certainly that would help.

 Here are the advantages to include revision number in the patch
 name:

- we would have the same ordering by name or by date
- it would be easier to refer to individual patch, say, when we
  need
to
refer to multiple patches when making a comment (e.g,,
 comparing
   revX
 with
revY, here are the pros and cons ...).
- when we create a new rev patch file before submitting, if we
 use
   the
same name as previous one, it would overwrite the previous one
- when we download patch files to the same directory, depending
 on
   the
order of downloading, the patches would possibly not appear in
 the
order
that they were submitted.

 Best regards,

 --Yongjun

 On Sat, Dec 13, 2014 at 10:54 AM, Konstantin Shvachko 
 shv.had...@gmail.com
 wrote:
 
  Hello guys,
 
  The problem here is not in a patch naming conventions, but in the
   jira
  default ordering schema for attachments.
  Mentioned it on several occasions. Currently attachments use
 sort
  by
 name
  sorting as the default. And it should be changed to sort by
 date.
Then
  you don't need any naming conventions to adjust to current
  sorting
  settings. You just see them in the order submitted and choose the
   last
 for
  a review or a commit.
 
  Does anybody have permissions  skills to change the default
 order
   type
 for
  attachments in the Jira?
 
  Thanks,
  --Konst
 
  On Thu, Dec 4, 2014 at 10:18 AM, Tsuyoshi OZAWA 
 ozawa.tsuyo...@gmail.com
  wrote:
  
   Thanks Yongjun and Harsh for updating Wiki!
  
   Thanks,
   - Tsuyoshi
  
   On Thu, Dec 4, 2014 at 9:43 AM, Yongjun Zhang 
  yzh...@cloudera.com
   
  wrote:
Thanks Harsh, I just made a change in
   
   
  https://wiki.apache.org/hadoop/HowToContribute

[jira] [Created] (HADOOP-11408) TestRetryCacheWithHA.testUpdatePipeline failed in trunk

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-11408:
--

 Summary: TestRetryCacheWithHA.testUpdatePipeline failed in trunk
 Key: HADOOP-11408
 URL: https://issues.apache.org/jira/browse/HADOOP-11408
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang


https://builds.apache.org/job/Hadoop-Hdfs-trunk/1974/testReport/

Error Message
{quote}
After waiting the operation updatePipeline still has not taken effect on NN yet
Stacktrace

java.lang.AssertionError: After waiting the operation updatePipeline still has 
not taken effect on NN yet
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1278)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline(TestRetryCacheWithHA.java:1176)
{quote}

Found by tool proposed in HADOOP-11045:

{quote}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j 
Hadoop-Hdfs-trunk -n 28 | tee bt.log
Recently FAILED builds in url: 
https://builds.apache.org//job/Hadoop-Hdfs-trunk
THERE ARE 4 builds (out of 6) that have failed tests in the past 28 days, 
as listed below:

===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1974/testReport (2014-12-15 
03:30:01)
Failed test: 
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
Failed test: 
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
Failed test: 
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1972/testReport (2014-12-13 
10:32:27)
Failed test: 
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1971/testReport (2014-12-13 
03:30:01)
Failed test: 
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1969/testReport (2014-12-11 
03:30:01)
Failed test: 
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
Failed test: 
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
Failed test: 
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization

Among 6 runs examined, all failed tests #failedRuns: testName:
3: 
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
2: org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
2: 
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
1: 
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization
{quote}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11408) TestRetryCacheWithHA.testUpdatePipeline failed in trunk

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang resolved HADOOP-11408.

Resolution: Duplicate

 TestRetryCacheWithHA.testUpdatePipeline failed in trunk
 ---

 Key: HADOOP-11408
 URL: https://issues.apache.org/jira/browse/HADOOP-11408
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yongjun Zhang

 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1974/testReport/
 Error Message
 {quote}
 After waiting the operation updatePipeline still has not taken effect on NN 
 yet
 Stacktrace
 java.lang.AssertionError: After waiting the operation updatePipeline still 
 has not taken effect on NN yet
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at 
 org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testClientRetryWithFailover(TestRetryCacheWithHA.java:1278)
   at 
 org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline(TestRetryCacheWithHA.java:1176)
 {quote}
 Found by tool proposed in HADOOP-11045:
 {quote}
 [yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j 
 Hadoop-Hdfs-trunk -n 5 | tee bt.log
 Recently FAILED builds in url: 
 https://builds.apache.org//job/Hadoop-Hdfs-trunk
 THERE ARE 4 builds (out of 6) that have failed tests in the past 5 days, 
 as listed below:
 ===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1974/testReport 
 (2014-12-15 03:30:01)
 Failed test: 
 org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
 Failed test: 
 org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
 Failed test: 
 org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
 ===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1972/testReport 
 (2014-12-13 10:32:27)
 Failed test: 
 org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
 ===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1971/testReport 
 (2014-12-13 03:30:01)
 Failed test: 
 org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
 ===https://builds.apache.org/job/Hadoop-Hdfs-trunk/1969/testReport 
 (2014-12-11 03:30:01)
 Failed test: 
 org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
 Failed test: 
 org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
 Failed test: 
 org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization
 Among 6 runs examined, all failed tests #failedRuns: testName:
 3: 
 org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA.testUpdatePipeline
 2: org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
 2: 
 org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
 1: 
 org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Hi Konst,

Thanks for the good suggestion, certainly that would help.

Here are the advantages to include revision number in the patch name:

   - we would have the same ordering by name or by date
   - it would be easier to refer to individual patch, say, when we need to
   refer to multiple patches when making a comment (e.g,, comparing revX with
   revY, here are the pros and cons ...).
   - when we create a new rev patch file before submitting, if we use the
   same name as previous one, it would overwrite the previous one
   - when we download patch files to the same directory, depending on the
   order of downloading, the patches would possibly not appear in the order
   that they were submitted.

Best regards,

--Yongjun

On Sat, Dec 13, 2014 at 10:54 AM, Konstantin Shvachko shv.had...@gmail.com
wrote:

 Hello guys,

 The problem here is not in a patch naming conventions, but in the jira
 default ordering schema for attachments.
 Mentioned it on several occasions. Currently attachments use sort by name
 sorting as the default. And it should be changed to sort by date. Then
 you don't need any naming conventions to adjust to current sorting
 settings. You just see them in the order submitted and choose the last for
 a review or a commit.

 Does anybody have permissions  skills to change the default order type for
 attachments in the Jira?

 Thanks,
 --Konst

 On Thu, Dec 4, 2014 at 10:18 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
 wrote:
 
  Thanks Yongjun and Harsh for updating Wiki!
 
  Thanks,
  - Tsuyoshi
 
  On Thu, Dec 4, 2014 at 9:43 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:
   Thanks Harsh, I just made a change in
  
   https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
  
   based on the discussion in this thread.
  
   --Yongjun
  
   On Wed, Dec 3, 2014 at 2:20 PM, Harsh J ha...@cloudera.com wrote:
  
   I've added you in as YongjunZhang. Please let me know if you are still
   unable to edit after a relogin.
  
   On Wed, Dec 3, 2014 at 1:43 AM, Yongjun Zhang yzh...@cloudera.com
  wrote:
Thanks Allen, Andrew and Tsuyoshi.
   
My wiki user name is YongjunZhang, I will appreciate it very much if
someone can give me the permission to edit the wiki pages. Thanks.
   
--Yongjun
   
On Tue, Dec 2, 2014 at 11:04 AM, Andrew Wang 
  andrew.w...@cloudera.com
wrote:
   
I just updated the wiki to say that the version number format is
   preferred.
Yongjun, if you email out your wiki username, someone (?) can give
  you
privs.
   
On Tue, Dec 2, 2014 at 10:16 AM, Allen Wittenauer 
 a...@altiscale.com
wrote:
   
 I think people forget we have a wiki that documents this and
 other
   things
 ...

 https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

 On Dec 2, 2014, at 10:01 AM, Tsuyoshi OZAWA 
  ozawa.tsuyo...@gmail.com
   
 wrote:

  jiraNameId.[branchName.]revisionNum.patch*
 
  +1 for this format. Thanks for starting the discussion,
 Yongjun.
 
  - Tsuyoshi
 
  On Tue, Dec 2, 2014 at 9:34 AM, Yongjun Zhang 
  yzh...@cloudera.com
 wrote:
  Thank you all for the feedback.
 
  About how many digits to use, I personally find it's not
  annoying
   to
 type
  one extra digit, but as long as we have the rev number, it
  achieves
the
  goal of identifying individual patch.
 
  About the rest of the name, as long as we keep it the same for
  the
same
  patch, it would work fine.
 
  This boils down to patch naming guideline:
 
  *jiraNameId.[branchName.]revisionNum.patch*
 
  - Example jiraNameId: HADOOP-1234, HDFS-4321
  - When the patch is targeted for trunk, then there is no
  need
   for
 the
  branchName portion, otherwise, specify the branchName
  accordingly.
 Example:
  branch1, branch2.
  - It's recommended to use three digits for revisionNum
 for
better
  sorting of different versions of patches.
 
  Would anyone who has the privilege please help to modify the
   following
 page
 
 
 http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
 
  accordingly?
 
  Thanks a lot.
 
  --Yongjun
 
  On Mon, Dec 1, 2014 at 10:22 AM, Colin McCabe 
   cmcc...@alumni.cmu.edu

  wrote:
 
  On Wed, Nov 26, 2014 at 2:58 PM, Karthik Kambatla 
ka...@cloudera.com
  wrote:
 
  Yongjun, thanks for starting this thread. I personally like
   Steve's
  suggestions, but think two digits should be enough.
 
  I propose we limit the restrictions to versioning the
 patches
   with
  version
  numbers and .patch extension. People have their own
  preferences
   for
 the
  rest of the name (e.g. MAPREDUCE, MapReduce, MR, mr, mapred)
  and
   I
 don't
  see a gain in forcing everyone to use one.
 
  Putting the suggestions (tight and loose

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Hi,

I wonder if anyone can help on resolving HADOOP-11320
https://issues.apache.org/jira/browse/HADOOP-11320 to increase timeout
for jenkins test of crossing-subproject patches?

Thanks a lot,

--Yongjun

On Tue, Dec 2, 2014 at 10:10 AM, Yongjun Zhang yzh...@cloudera.com wrote:

 Hi,

 Thank you all for the input.

 https://issues.apache.org/jira/browse/HADOOP-11320

 was created for this issue. Welcome to give your further comments there.

 Best,

 --Yongjun

 On Tue, Nov 25, 2014 at 10:26 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

 +1 for increasing the test timeout for tests spanning multiple
 sub-projects.

 I can see the value in what Steve L. suggested... if you make a major
 change that touches a particular subproject, you should try to get the
 approval of a committer who knows that subproject.  But I don't think that
 forcing artificial patch splits is the way to do this...  There are also
 some patches that are completely mechanical and don't really require the
 involvement of YARN / HDFS committer, even if they change that project.
 For example, fixing a misspelling in the name of a hadoop-common API.

 Colin

 On Tue, Nov 25, 2014 at 8:45 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:

  Thanks all for the feedback. To summarize (and I have a suggestion at
 the
  end of this email), there are two scenarios:
 
 1. A change that span multiple *bigger* projects. r.g. hadoop, hbase.
 2. A change that span multiple *sub* projects* within hadoop, e.g.,
 common, hdfs, yarn
 
  For 1, it's required for the change to be backward compatible, thus
  splitting change for multiple *bigger* projects is a must.
 
  For 2, there are two sub types,
 
 - 2.1 those changes that can be made within hadoop sub-projects, and
 there is no external impact
 - 2.2 those changes that have external impact, that is, the changes
 involve adding new APIs and marking old API deprecated, and
  corresponding
 changes in other *bigger* projects will have to be made
 independently.
  *But
 the changes within hadoop subjects can still be done altogether.*
 
  I think (Please correct me if I'm wrong):
 
 - What Colin referred to is 2.1 and changes within hadoop
 sub-subjects
 for 2.2;
 - Steve's not for changes across hadoop-common and hdfs, or
 hadoop-common and yarn means 2.1, Steve's  changes that only
 span hdfs-and-yarn would be fairly doubtful too. implies his doubt
 of
 existence of 2.1.
 
  For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
  option of making the change across all hadoop sub-projects altogether,
 to
  save the multiple steps Colin referred to.
 
  If this option is feasible, should we consider increasing the jenkins
  timeout for this kind of changes (I mean making the timeout adjustable,
 if
  it's for single sub-project, use the old timeout; otherwise, increase
  accordingly)  so that we have at least this option when needed?
 
  Thanks.
 
  --Yongjun
 
 
  On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran ste...@hortonworks.com
 
  wrote:
 
   On 25 November 2014 at 00:58, Bernd Eckenfels e...@zusammenkunft.net
 
   wrote:
  
Hello,
   
Am Mon, 24 Nov 2014 16:16:00 -0800
schrieb Colin McCabe cmcc...@alumni.cmu.edu:
   
 Conceptually, I think it's important to support patches that
 modify
 multiple sub-projects.  Otherwise refactoring things in common
 becomes a multi-step process.
   
This might be rather philosophical (and I dont want to argue the
 need
to have the patch infrastructure work for the multi-project case),
howevere if a multi-project change cannot be applied in multiple
 steps
it is probably also not safe at runtime (unless the multiple
 projects
belong to a single instance/artifact). And then beeing forced to
commit/compile/test in multiple steps actually increases the
dependencies topology.
   
  
   +1 for changes that span, say hadoop and hbase. but not for changes
  across
   hadoop-common and hdfs, or hadoop-common and yarn. changes that only
 span
   hdfs-and-yarn would be fairly doubtful too.
  
   there is a dependency graph in hadoop's own jars —and cross module
 (not
   cross project) changes do need to happen.
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is
 confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Many thanks to Ted Yu, Steve Loughran and Andrew Wang for replying in the
jira and Steve/Andrew for making the related changes!

--Yongjun

On Thu, Dec 11, 2014 at 12:41 PM, Yongjun Zhang yzh...@cloudera.com wrote:

 Hi,

 I wonder if anyone can help on resolving HADOOP-11320
 https://issues.apache.org/jira/browse/HADOOP-11320 to increase timeout
 for jenkins test of crossing-subproject patches?

 Thanks a lot,

 --Yongjun

 On Tue, Dec 2, 2014 at 10:10 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:

 Hi,

 Thank you all for the input.

 https://issues.apache.org/jira/browse/HADOOP-11320

 was created for this issue. Welcome to give your further comments there.

 Best,

 --Yongjun

 On Tue, Nov 25, 2014 at 10:26 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

 +1 for increasing the test timeout for tests spanning multiple
 sub-projects.

 I can see the value in what Steve L. suggested... if you make a major
 change that touches a particular subproject, you should try to get the
 approval of a committer who knows that subproject.  But I don't think
 that
 forcing artificial patch splits is the way to do this...  There are also
 some patches that are completely mechanical and don't really require the
 involvement of YARN / HDFS committer, even if they change that project.
 For example, fixing a misspelling in the name of a hadoop-common API.

 Colin

 On Tue, Nov 25, 2014 at 8:45 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:

  Thanks all for the feedback. To summarize (and I have a suggestion at
 the
  end of this email), there are two scenarios:
 
 1. A change that span multiple *bigger* projects. r.g. hadoop,
 hbase.
 2. A change that span multiple *sub* projects* within hadoop, e.g.,
 common, hdfs, yarn
 
  For 1, it's required for the change to be backward compatible, thus
  splitting change for multiple *bigger* projects is a must.
 
  For 2, there are two sub types,
 
 - 2.1 those changes that can be made within hadoop sub-projects, and
 there is no external impact
 - 2.2 those changes that have external impact, that is, the changes
 involve adding new APIs and marking old API deprecated, and
  corresponding
 changes in other *bigger* projects will have to be made
 independently.
  *But
 the changes within hadoop subjects can still be done altogether.*
 
  I think (Please correct me if I'm wrong):
 
 - What Colin referred to is 2.1 and changes within hadoop
 sub-subjects
 for 2.2;
 - Steve's not for changes across hadoop-common and hdfs, or
 hadoop-common and yarn means 2.1, Steve's  changes that only
 span hdfs-and-yarn would be fairly doubtful too. implies his doubt
 of
 existence of 2.1.
 
  For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
  option of making the change across all hadoop sub-projects altogether,
 to
  save the multiple steps Colin referred to.
 
  If this option is feasible, should we consider increasing the jenkins
  timeout for this kind of changes (I mean making the timeout
 adjustable, if
  it's for single sub-project, use the old timeout; otherwise, increase
  accordingly)  so that we have at least this option when needed?
 
  Thanks.
 
  --Yongjun
 
 
  On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran 
 ste...@hortonworks.com
  wrote:
 
   On 25 November 2014 at 00:58, Bernd Eckenfels 
 e...@zusammenkunft.net
   wrote:
  
Hello,
   
Am Mon, 24 Nov 2014 16:16:00 -0800
schrieb Colin McCabe cmcc...@alumni.cmu.edu:
   
 Conceptually, I think it's important to support patches that
 modify
 multiple sub-projects.  Otherwise refactoring things in common
 becomes a multi-step process.
   
This might be rather philosophical (and I dont want to argue the
 need
to have the patch infrastructure work for the multi-project case),
howevere if a multi-project change cannot be applied in multiple
 steps
it is probably also not safe at runtime (unless the multiple
 projects
belong to a single instance/artifact). And then beeing forced to
commit/compile/test in multiple steps actually increases the
dependencies topology.
   
  
   +1 for changes that span, say hadoop and hbase. but not for changes
  across
   hadoop-common and hdfs, or hadoop-common and yarn. changes that only
 span
   hdfs-and-yarn would be fairly doubtful too.
  
   there is a dependency graph in hadoop's own jars —and cross module
 (not
   cross project) changes do need to happen.
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is
 confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby
 notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Sorry my bad, I named Andrew Wang for Andrew Bayer in my last mail, both of
them helped anyways:-) So thanks to all for the help on this matter!

--Yongjun

On Thu, Dec 11, 2014 at 3:38 PM, Yongjun Zhang yzh...@cloudera.com wrote:

 Many thanks to Ted Yu, Steve Loughran and Andrew Wang for replying in the
 jira and Steve/Andrew for making the related changes!

 --Yongjun

 On Thu, Dec 11, 2014 at 12:41 PM, Yongjun Zhang yzh...@cloudera.com
 wrote:

 Hi,

 I wonder if anyone can help on resolving HADOOP-11320
 https://issues.apache.org/jira/browse/HADOOP-11320 to increase timeout
 for jenkins test of crossing-subproject patches?

 Thanks a lot,

 --Yongjun

 On Tue, Dec 2, 2014 at 10:10 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:

 Hi,

 Thank you all for the input.

 https://issues.apache.org/jira/browse/HADOOP-11320

 was created for this issue. Welcome to give your further comments there.

 Best,

 --Yongjun

 On Tue, Nov 25, 2014 at 10:26 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

 +1 for increasing the test timeout for tests spanning multiple
 sub-projects.

 I can see the value in what Steve L. suggested... if you make a major
 change that touches a particular subproject, you should try to get the
 approval of a committer who knows that subproject.  But I don't think
 that
 forcing artificial patch splits is the way to do this...  There are also
 some patches that are completely mechanical and don't really require the
 involvement of YARN / HDFS committer, even if they change that project.
 For example, fixing a misspelling in the name of a hadoop-common API.

 Colin

 On Tue, Nov 25, 2014 at 8:45 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:

  Thanks all for the feedback. To summarize (and I have a suggestion at
 the
  end of this email), there are two scenarios:
 
 1. A change that span multiple *bigger* projects. r.g. hadoop,
 hbase.
 2. A change that span multiple *sub* projects* within hadoop, e.g.,
 common, hdfs, yarn
 
  For 1, it's required for the change to be backward compatible, thus
  splitting change for multiple *bigger* projects is a must.
 
  For 2, there are two sub types,
 
 - 2.1 those changes that can be made within hadoop sub-projects,
 and
 there is no external impact
 - 2.2 those changes that have external impact, that is, the changes
 involve adding new APIs and marking old API deprecated, and
  corresponding
 changes in other *bigger* projects will have to be made
 independently.
  *But
 the changes within hadoop subjects can still be done altogether.*
 
  I think (Please correct me if I'm wrong):
 
 - What Colin referred to is 2.1 and changes within hadoop
 sub-subjects
 for 2.2;
 - Steve's not for changes across hadoop-common and hdfs, or
 hadoop-common and yarn means 2.1, Steve's  changes that only
 span hdfs-and-yarn would be fairly doubtful too. implies his
 doubt of
 existence of 2.1.
 
  For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
  option of making the change across all hadoop sub-projects
 altogether, to
  save the multiple steps Colin referred to.
 
  If this option is feasible, should we consider increasing the jenkins
  timeout for this kind of changes (I mean making the timeout
 adjustable, if
  it's for single sub-project, use the old timeout; otherwise, increase
  accordingly)  so that we have at least this option when needed?
 
  Thanks.
 
  --Yongjun
 
 
  On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran 
 ste...@hortonworks.com
  wrote:
 
   On 25 November 2014 at 00:58, Bernd Eckenfels 
 e...@zusammenkunft.net
   wrote:
  
Hello,
   
Am Mon, 24 Nov 2014 16:16:00 -0800
schrieb Colin McCabe cmcc...@alumni.cmu.edu:
   
 Conceptually, I think it's important to support patches that
 modify
 multiple sub-projects.  Otherwise refactoring things in common
 becomes a multi-step process.
   
This might be rather philosophical (and I dont want to argue the
 need
to have the patch infrastructure work for the multi-project case),
howevere if a multi-project change cannot be applied in multiple
 steps
it is probably also not safe at runtime (unless the multiple
 projects
belong to a single instance/artifact). And then beeing forced to
commit/compile/test in multiple steps actually increases the
dependencies topology.
   
  
   +1 for changes that span, say hadoop and hbase. but not for changes
  across
   hadoop-common and hdfs, or hadoop-common and yarn. changes that
 only span
   hdfs-and-yarn would be fairly doubtful too.
  
   there is a dependency graph in hadoop's own jars —and cross module
 (not
   cross project) changes do need to happen.
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is
 confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Thanks Harsh, I just made a change in

https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

based on the discussion in this thread.

--Yongjun

On Wed, Dec 3, 2014 at 2:20 PM, Harsh J ha...@cloudera.com wrote:

 I've added you in as YongjunZhang. Please let me know if you are still
 unable to edit after a relogin.

 On Wed, Dec 3, 2014 at 1:43 AM, Yongjun Zhang yzh...@cloudera.com wrote:
  Thanks Allen, Andrew and Tsuyoshi.
 
  My wiki user name is YongjunZhang, I will appreciate it very much if
  someone can give me the permission to edit the wiki pages. Thanks.
 
  --Yongjun
 
  On Tue, Dec 2, 2014 at 11:04 AM, Andrew Wang andrew.w...@cloudera.com
  wrote:
 
  I just updated the wiki to say that the version number format is
 preferred.
  Yongjun, if you email out your wiki username, someone (?) can give you
  privs.
 
  On Tue, Dec 2, 2014 at 10:16 AM, Allen Wittenauer a...@altiscale.com
  wrote:
 
   I think people forget we have a wiki that documents this and other
 things
   ...
  
   https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
  
   On Dec 2, 2014, at 10:01 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
 
   wrote:
  
jiraNameId.[branchName.]revisionNum.patch*
   
+1 for this format. Thanks for starting the discussion, Yongjun.
   
- Tsuyoshi
   
On Tue, Dec 2, 2014 at 9:34 AM, Yongjun Zhang yzh...@cloudera.com
   wrote:
Thank you all for the feedback.
   
About how many digits to use, I personally find it's not annoying
 to
   type
one extra digit, but as long as we have the rev number, it achieves
  the
goal of identifying individual patch.
   
About the rest of the name, as long as we keep it the same for the
  same
patch, it would work fine.
   
This boils down to patch naming guideline:
   
*jiraNameId.[branchName.]revisionNum.patch*
   
- Example jiraNameId: HADOOP-1234, HDFS-4321
- When the patch is targeted for trunk, then there is no need
 for
   the
branchName portion, otherwise, specify the branchName accordingly.
   Example:
branch1, branch2.
- It's recommended to use three digits for revisionNum for
  better
sorting of different versions of patches.
   
Would anyone who has the privilege please help to modify the
 following
   page
   
http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
   
accordingly?
   
Thanks a lot.
   
--Yongjun
   
On Mon, Dec 1, 2014 at 10:22 AM, Colin McCabe 
 cmcc...@alumni.cmu.edu
  
wrote:
   
On Wed, Nov 26, 2014 at 2:58 PM, Karthik Kambatla 
  ka...@cloudera.com
wrote:
   
Yongjun, thanks for starting this thread. I personally like
 Steve's
suggestions, but think two digits should be enough.
   
I propose we limit the restrictions to versioning the patches
 with
version
numbers and .patch extension. People have their own preferences
 for
   the
rest of the name (e.g. MAPREDUCE, MapReduce, MR, mr, mapred) and
 I
   don't
see a gain in forcing everyone to use one.
   
Putting the suggestions (tight and loose) on the wiki would help
 new
contributors as well.
   
   
+1
   
best,
Colin
   
   
On Wed, Nov 26, 2014 at 2:43 PM, Eric Payne
erichadoo...@yahoo.com.invalid
   
wrote:
   
+1.The different color for newest patch doesn't work very
 well if
   you
are color blind, so I do appreciate a revision number in the
 name.
   
 From: Yongjun Zhang yzh...@cloudera.com
To: common-dev@hadoop.apache.org
Sent: Tuesday, November 25, 2014 11:37 PM
Subject: Re: a friendly suggestion for developers when uploading
patches
   
Thanks Harsh for the info and Andrew for sharing the script. It
  looks
that
the script is intelligent enough to pick the latest attachment
 even
   if
all
attachments have the same name.
   
Yet, I hope we use the following as the guideline for patch
 names:
   
*projectName*-*jiraNum*-*revNum*.patch
   
   
So we can easily identify individual patch revs.
   
Thanks.
   
--Yongjun
   
On Tue, Nov 25, 2014 at 5:54 PM, Andrew Wang 
   andrew.w...@cloudera.com
   
wrote:
   
This might be a good time to mention my fetch-patch script, I
 use
  it
to
easily download the latest attachment on a jira:
   
   
 https://github.com/umbrant/dotfiles/blob/master/bin/fetch-patch
   
On Tue, Nov 25, 2014 at 5:44 PM, Harsh J ha...@cloudera.com
   wrote:
   
For the same filename, you can observe also that the JIRA
 colors
the
latest one to be different than the older ones automatically -
  this
is
what I rely on.
   
On Sat, Nov 22, 2014 at 12:36 AM, Yongjun Zhang 
yzh...@cloudera.com
   
wrote:
Hi,
   
When I look at patches uploaded to jiras, from time to time I
notice
that
different revisions of the patch is uploaded with the same
 patch
file
name,
some time for quite a few times

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Thank you all for the feedback.

About how many digits to use, I personally find it's not annoying to type
one extra digit, but as long as we have the rev number, it achieves the
goal of identifying individual patch.

About the rest of the name, as long as we keep it the same for the same
patch, it would work fine.

This boils down to patch naming guideline:

*jiraNameId.[branchName.]revisionNum.patch*

 - Example jiraNameId: HADOOP-1234, HDFS-4321
 - When the patch is targeted for trunk, then there is no need for the
branchName portion, otherwise, specify the branchName accordingly. Example:
branch1, branch2.
 - It's recommended to use three digits for revisionNum for better
sorting of different versions of patches.

Would anyone who has the privilege please help to modify the following page

http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch

accordingly?

Thanks a lot.

--Yongjun

On Mon, Dec 1, 2014 at 10:22 AM, Colin McCabe cmcc...@alumni.cmu.edu
wrote:

 On Wed, Nov 26, 2014 at 2:58 PM, Karthik Kambatla ka...@cloudera.com
 wrote:

  Yongjun, thanks for starting this thread. I personally like Steve's
  suggestions, but think two digits should be enough.
 
  I propose we limit the restrictions to versioning the patches with
 version
  numbers and .patch extension. People have their own preferences for the
  rest of the name (e.g. MAPREDUCE, MapReduce, MR, mr, mapred) and I don't
  see a gain in forcing everyone to use one.
 
  Putting the suggestions (tight and loose) on the wiki would help new
  contributors as well.
 
 
 +1

 best,
 Colin


  On Wed, Nov 26, 2014 at 2:43 PM, Eric Payne
 erichadoo...@yahoo.com.invalid
  
  wrote:
 
   +1.The different color for newest patch doesn't work very well if you
   are color blind, so I do appreciate a revision number in the name.
  
 From: Yongjun Zhang yzh...@cloudera.com
To: common-dev@hadoop.apache.org
Sent: Tuesday, November 25, 2014 11:37 PM
Subject: Re: a friendly suggestion for developers when uploading
 patches
  
   Thanks Harsh for the info and Andrew for sharing the script. It looks
  that
   the script is intelligent enough to pick the latest attachment even if
  all
   attachments have the same name.
  
   Yet, I hope we use the following as the guideline for patch names:
  
   *projectName*-*jiraNum*-*revNum*.patch
  
  
   So we can easily identify individual patch revs.
  
   Thanks.
  
   --Yongjun
  
   On Tue, Nov 25, 2014 at 5:54 PM, Andrew Wang andrew.w...@cloudera.com
 
   wrote:
  
This might be a good time to mention my fetch-patch script, I use it
 to
easily download the latest attachment on a jira:
   
https://github.com/umbrant/dotfiles/blob/master/bin/fetch-patch
   
On Tue, Nov 25, 2014 at 5:44 PM, Harsh J ha...@cloudera.com wrote:
   
 For the same filename, you can observe also that the JIRA colors
 the
 latest one to be different than the older ones automatically - this
  is
 what I rely on.

 On Sat, Nov 22, 2014 at 12:36 AM, Yongjun Zhang 
 yzh...@cloudera.com
  
 wrote:
  Hi,
 
  When I look at patches uploaded to jiras, from time to time I
  notice
that
  different revisions of the patch is uploaded with the same patch
  file
 name,
  some time for quite a few times. It's confusing which is which.
 
  I'd suggest that as a guideline, we do the following when
  uploading a
 patch:
 
 - include a revision number in the patch file name.A
 - include a comment, stating that a new patch is uploaded,
   including
 the
 revision number of the patch in the comment.
 
  This way, it's easier to refer to a specific version of a patch,
  and
   to
  know which patch a comment is made about.
 
  Hope that makes sense to you.
 
  Thanks.
 
  --Yongjun



 --
 Harsh J

   
  
  
  
  
 

[jira] [Reopened] (HADOOP-11320) Submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang reopened HADOOP-11320:


 Submitting a hadoop patch doesn't trigger jenkins test run
 --

 Key: HADOOP-11320
 URL: https://issues.apache.org/jira/browse/HADOOP-11320
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Yongjun Zhang
 Attachments: HADOOP-11293.003.patch


 See details in INFRA-8655.
 Per [~abayer] and [~cnauroth]'s feedback there , I'm creating this jira to 
 investigate the possible bug in dev-support/test-patch.sh script.
 Thanks Andrew and Chris.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Hi,

Thank you all for the input.

https://issues.apache.org/jira/browse/HADOOP-11320

was created for this issue. Welcome to give your further comments there.

Best,

--Yongjun

On Tue, Nov 25, 2014 at 10:26 PM, Colin McCabe cmcc...@alumni.cmu.edu
wrote:

 +1 for increasing the test timeout for tests spanning multiple
 sub-projects.

 I can see the value in what Steve L. suggested... if you make a major
 change that touches a particular subproject, you should try to get the
 approval of a committer who knows that subproject.  But I don't think that
 forcing artificial patch splits is the way to do this...  There are also
 some patches that are completely mechanical and don't really require the
 involvement of YARN / HDFS committer, even if they change that project.
 For example, fixing a misspelling in the name of a hadoop-common API.

 Colin

 On Tue, Nov 25, 2014 at 8:45 AM, Yongjun Zhang yzh...@cloudera.com
 wrote:

  Thanks all for the feedback. To summarize (and I have a suggestion at the
  end of this email), there are two scenarios:
 
 1. A change that span multiple *bigger* projects. r.g. hadoop, hbase.
 2. A change that span multiple *sub* projects* within hadoop, e.g.,
 common, hdfs, yarn
 
  For 1, it's required for the change to be backward compatible, thus
  splitting change for multiple *bigger* projects is a must.
 
  For 2, there are two sub types,
 
 - 2.1 those changes that can be made within hadoop sub-projects, and
 there is no external impact
 - 2.2 those changes that have external impact, that is, the changes
 involve adding new APIs and marking old API deprecated, and
  corresponding
 changes in other *bigger* projects will have to be made independently.
  *But
 the changes within hadoop subjects can still be done altogether.*
 
  I think (Please correct me if I'm wrong):
 
 - What Colin referred to is 2.1 and changes within hadoop sub-subjects
 for 2.2;
 - Steve's not for changes across hadoop-common and hdfs, or
 hadoop-common and yarn means 2.1, Steve's  changes that only
 span hdfs-and-yarn would be fairly doubtful too. implies his doubt of
 existence of 2.1.
 
  For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
  option of making the change across all hadoop sub-projects altogether, to
  save the multiple steps Colin referred to.
 
  If this option is feasible, should we consider increasing the jenkins
  timeout for this kind of changes (I mean making the timeout adjustable,
 if
  it's for single sub-project, use the old timeout; otherwise, increase
  accordingly)  so that we have at least this option when needed?
 
  Thanks.
 
  --Yongjun
 
 
  On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran ste...@hortonworks.com
  wrote:
 
   On 25 November 2014 at 00:58, Bernd Eckenfels e...@zusammenkunft.net
   wrote:
  
Hello,
   
Am Mon, 24 Nov 2014 16:16:00 -0800
schrieb Colin McCabe cmcc...@alumni.cmu.edu:
   
 Conceptually, I think it's important to support patches that modify
 multiple sub-projects.  Otherwise refactoring things in common
 becomes a multi-step process.
   
This might be rather philosophical (and I dont want to argue the need
to have the patch infrastructure work for the multi-project case),
howevere if a multi-project change cannot be applied in multiple
 steps
it is probably also not safe at runtime (unless the multiple projects
belong to a single instance/artifact). And then beeing forced to
commit/compile/test in multiple steps actually increases the
dependencies topology.
   
  
   +1 for changes that span, say hadoop and hbase. but not for changes
  across
   hadoop-common and hdfs, or hadoop-common and yarn. changes that only
 span
   hdfs-and-yarn would be fairly doubtful too.
  
   there is a dependency graph in hadoop's own jars —and cross module (not
   cross project) changes do need to happen.
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Thanks Allen, Andrew and Tsuyoshi.

My wiki user name is YongjunZhang, I will appreciate it very much if
someone can give me the permission to edit the wiki pages. Thanks.

--Yongjun

On Tue, Dec 2, 2014 at 11:04 AM, Andrew Wang andrew.w...@cloudera.com
wrote:

 I just updated the wiki to say that the version number format is preferred.
 Yongjun, if you email out your wiki username, someone (?) can give you
 privs.

 On Tue, Dec 2, 2014 at 10:16 AM, Allen Wittenauer a...@altiscale.com
 wrote:

  I think people forget we have a wiki that documents this and other things
  ...
 
  https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
 
  On Dec 2, 2014, at 10:01 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
  wrote:
 
   jiraNameId.[branchName.]revisionNum.patch*
  
   +1 for this format. Thanks for starting the discussion, Yongjun.
  
   - Tsuyoshi
  
   On Tue, Dec 2, 2014 at 9:34 AM, Yongjun Zhang yzh...@cloudera.com
  wrote:
   Thank you all for the feedback.
  
   About how many digits to use, I personally find it's not annoying to
  type
   one extra digit, but as long as we have the rev number, it achieves
 the
   goal of identifying individual patch.
  
   About the rest of the name, as long as we keep it the same for the
 same
   patch, it would work fine.
  
   This boils down to patch naming guideline:
  
   *jiraNameId.[branchName.]revisionNum.patch*
  
   - Example jiraNameId: HADOOP-1234, HDFS-4321
   - When the patch is targeted for trunk, then there is no need for
  the
   branchName portion, otherwise, specify the branchName accordingly.
  Example:
   branch1, branch2.
   - It's recommended to use three digits for revisionNum for
 better
   sorting of different versions of patches.
  
   Would anyone who has the privilege please help to modify the following
  page
  
   http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
  
   accordingly?
  
   Thanks a lot.
  
   --Yongjun
  
   On Mon, Dec 1, 2014 at 10:22 AM, Colin McCabe cmcc...@alumni.cmu.edu
 
   wrote:
  
   On Wed, Nov 26, 2014 at 2:58 PM, Karthik Kambatla 
 ka...@cloudera.com
   wrote:
  
   Yongjun, thanks for starting this thread. I personally like Steve's
   suggestions, but think two digits should be enough.
  
   I propose we limit the restrictions to versioning the patches with
   version
   numbers and .patch extension. People have their own preferences for
  the
   rest of the name (e.g. MAPREDUCE, MapReduce, MR, mr, mapred) and I
  don't
   see a gain in forcing everyone to use one.
  
   Putting the suggestions (tight and loose) on the wiki would help new
   contributors as well.
  
  
   +1
  
   best,
   Colin
  
  
   On Wed, Nov 26, 2014 at 2:43 PM, Eric Payne
   erichadoo...@yahoo.com.invalid
  
   wrote:
  
   +1.The different color for newest patch doesn't work very well if
  you
   are color blind, so I do appreciate a revision number in the name.
  
From: Yongjun Zhang yzh...@cloudera.com
   To: common-dev@hadoop.apache.org
   Sent: Tuesday, November 25, 2014 11:37 PM
   Subject: Re: a friendly suggestion for developers when uploading
   patches
  
   Thanks Harsh for the info and Andrew for sharing the script. It
 looks
   that
   the script is intelligent enough to pick the latest attachment even
  if
   all
   attachments have the same name.
  
   Yet, I hope we use the following as the guideline for patch names:
  
   *projectName*-*jiraNum*-*revNum*.patch
  
  
   So we can easily identify individual patch revs.
  
   Thanks.
  
   --Yongjun
  
   On Tue, Nov 25, 2014 at 5:54 PM, Andrew Wang 
  andrew.w...@cloudera.com
  
   wrote:
  
   This might be a good time to mention my fetch-patch script, I use
 it
   to
   easily download the latest attachment on a jira:
  
   https://github.com/umbrant/dotfiles/blob/master/bin/fetch-patch
  
   On Tue, Nov 25, 2014 at 5:44 PM, Harsh J ha...@cloudera.com
  wrote:
  
   For the same filename, you can observe also that the JIRA colors
   the
   latest one to be different than the older ones automatically -
 this
   is
   what I rely on.
  
   On Sat, Nov 22, 2014 at 12:36 AM, Yongjun Zhang 
   yzh...@cloudera.com
  
   wrote:
   Hi,
  
   When I look at patches uploaded to jiras, from time to time I
   notice
   that
   different revisions of the patch is uploaded with the same patch
   file
   name,
   some time for quite a few times. It's confusing which is which.
  
   I'd suggest that as a guideline, we do the following when
   uploading a
   patch:
  
 - include a revision number in the patch file name.A
 - include a comment, stating that a new patch is uploaded,
   including
   the
 revision number of the patch in the comment.
  
   This way, it's easier to refer to a specific version of a patch,
   and
   to
   know which patch a comment is made about.
  
   Hope that makes sense to you.
  
   Thanks.
  
   --Yongjun
  
  
  
   --
   Harsh J
  
  
  
  
  
  
  
  
  
  
  
   --
   - Tsuyoshi
 
 

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Thanks all for the feedback. To summarize (and I have a suggestion at the
end of this email), there are two scenarios:

   1. A change that span multiple *bigger* projects. r.g. hadoop, hbase.
   2. A change that span multiple *sub* projects* within hadoop, e.g.,
   common, hdfs, yarn

For 1, it's required for the change to be backward compatible, thus
splitting change for multiple *bigger* projects is a must.

For 2, there are two sub types,

   - 2.1 those changes that can be made within hadoop sub-projects, and
   there is no external impact
   - 2.2 those changes that have external impact, that is, the changes
   involve adding new APIs and marking old API deprecated, and corresponding
   changes in other *bigger* projects will have to be made independently. *But
   the changes within hadoop subjects can still be done altogether.*

I think (Please correct me if I'm wrong):

   - What Colin referred to is 2.1 and changes within hadoop sub-subjects
   for 2.2;
   - Steve's not for changes across hadoop-common and hdfs, or
   hadoop-common and yarn means 2.1, Steve's  changes that only
   span hdfs-and-yarn would be fairly doubtful too. implies his doubt of
   existence of 2.1.

For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
option of making the change across all hadoop sub-projects altogether, to
save the multiple steps Colin referred to.

If this option is feasible, should we consider increasing the jenkins
timeout for this kind of changes (I mean making the timeout adjustable, if
it's for single sub-project, use the old timeout; otherwise, increase
accordingly)  so that we have at least this option when needed?

Thanks.

--Yongjun


On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran ste...@hortonworks.com
wrote:

 On 25 November 2014 at 00:58, Bernd Eckenfels e...@zusammenkunft.net
 wrote:

  Hello,
 
  Am Mon, 24 Nov 2014 16:16:00 -0800
  schrieb Colin McCabe cmcc...@alumni.cmu.edu:
 
   Conceptually, I think it's important to support patches that modify
   multiple sub-projects.  Otherwise refactoring things in common
   becomes a multi-step process.
 
  This might be rather philosophical (and I dont want to argue the need
  to have the patch infrastructure work for the multi-project case),
  howevere if a multi-project change cannot be applied in multiple steps
  it is probably also not safe at runtime (unless the multiple projects
  belong to a single instance/artifact). And then beeing forced to
  commit/compile/test in multiple steps actually increases the
  dependencies topology.
 

 +1 for changes that span, say hadoop and hbase. but not for changes across
 hadoop-common and hdfs, or hadoop-common and yarn. changes that only span
 hdfs-and-yarn would be fairly doubtful too.

 there is a dependency graph in hadoop's own jars —and cross module (not
 cross project) changes do need to happen.

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Thanks Harsh for the info and Andrew for sharing the script. It looks that
the script is intelligent enough to pick the latest attachment even if all
attachments have the same name.

Yet, I hope we use the following as the guideline for patch names:

*projectName*-*jiraNum*-*revNum*.patch


So we can easily identify individual patch revs.

Thanks.

--Yongjun

On Tue, Nov 25, 2014 at 5:54 PM, Andrew Wang andrew.w...@cloudera.com
wrote:

 This might be a good time to mention my fetch-patch script, I use it to
 easily download the latest attachment on a jira:

 https://github.com/umbrant/dotfiles/blob/master/bin/fetch-patch

 On Tue, Nov 25, 2014 at 5:44 PM, Harsh J ha...@cloudera.com wrote:

  For the same filename, you can observe also that the JIRA colors the
  latest one to be different than the older ones automatically - this is
  what I rely on.
 
  On Sat, Nov 22, 2014 at 12:36 AM, Yongjun Zhang yzh...@cloudera.com
  wrote:
   Hi,
  
   When I look at patches uploaded to jiras, from time to time I notice
 that
   different revisions of the patch is uploaded with the same patch file
  name,
   some time for quite a few times. It's confusing which is which.
  
   I'd suggest that as a guideline, we do the following when uploading a
  patch:
  
  - include a revision number in the patch file name.A
  - include a comment, stating that a new patch is uploaded, including
  the
  revision number of the patch in the comment.
  
   This way, it's easier to refer to a specific version of a patch, and to
   know which patch a comment is made about.
  
   Hope that makes sense to you.
  
   Thanks.
  
   --Yongjun
 
 
 
  --
  Harsh J
 

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Hi Steve,

Thanks for the good suggestion.

I like the idea to have even a more specific guideline for patch file
naming, and I agree using 3-digit is a good choice here:

*projectName*-*jiraNum*-*revNum*.patch

where revNum is 3-digit, in the format of 001, 002, ..., 010, 011, ...

Thanks.

--Yongjun

On Sat, Nov 22, 2014 at 10:24 AM, Steve Loughran ste...@hortonworks.com
wrote:

 can we do HADOOP--001.patch

 with the 001 being the revision.

 -That numbering scheme guarantees listing order in directories c
 -having .patch come after ensures that those people who have .patch bound
 in their browser to a text editor (e.g. textmate) can view the patch with
 ease

 I know having a 3 digit number is pessimistic -I've never got past 70+, but
 you never know

 For anyone doing patches off their own repo, I'd recommend tagging the
 commit with the same revision number —but that 's just a personal choice



 On 21 November 2014 at 19:10, Ted Yu yuzhih...@gmail.com wrote:

  bq. include a revision number in the patch file name
 
  +1
 
  On Fri, Nov 21, 2014 at 11:06 AM, Yongjun Zhang yzh...@cloudera.com
  wrote:
 
   Hi,
  
   When I look at patches uploaded to jiras, from time to time I notice
 that
   different revisions of the patch is uploaded with the same patch file
  name,
   some time for quite a few times. It's confusing which is which.
  
   I'd suggest that as a guideline, we do the following when uploading a
   patch:
  
  - include a revision number in the patch file name.A
  - include a comment, stating that a new patch is uploaded, including
  the
  revision number of the patch in the comment.
  
   This way, it's easier to refer to a specific version of a patch, and to
   know which patch a comment is made about.
  
   Hope that makes sense to you.
  
   Thanks.
  
   --Yongjun
  
 

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: a friendly suggestion for developers when uploading patches

[Yongjun Zhang]

Hi Ted,

Thanks for sharing, this is a good tool to view the patch.

For those who haven't tried: once you install it, then when you mouse-click
on an uploaded patch in Chrome, Chrome will render the patch as if you are
diffing the patch against the base side by side, with left side showing the
base, and right side showing ALL the code change made by the patch, on ONE
screen.

I sometimes just apply the patch to the code base, and use git difftool to
view, but this way I can only view one file at a time, so I have to click
again and again to go through each file changed.

--Yongjun

On Sat, Nov 22, 2014 at 11:04 AM, Ted Yu yuzhih...@gmail.com wrote:

 For patch viewing, David Deng has a Chrome extension for rendering patch on
 JIRA.
 See this thread:
 http://search-hadoop.com/m/DHED4LHEYI

 FYI

 On Sat, Nov 22, 2014 at 10:24 AM, Steve Loughran ste...@hortonworks.com
 wrote:

  can we do HADOOP--001.patch
 
  with the 001 being the revision.
 
  -That numbering scheme guarantees listing order in directories c
  -having .patch come after ensures that those people who have .patch bound
  in their browser to a text editor (e.g. textmate) can view the patch with
  ease
 
  I know having a 3 digit number is pessimistic -I've never got past 70+,
 but
  you never know
 
  For anyone doing patches off their own repo, I'd recommend tagging the
  commit with the same revision number —but that 's just a personal choice
 
 
 
  On 21 November 2014 at 19:10, Ted Yu yuzhih...@gmail.com wrote:
 
   bq. include a revision number in the patch file name
  
   +1
  
   On Fri, Nov 21, 2014 at 11:06 AM, Yongjun Zhang yzh...@cloudera.com
   wrote:
  
Hi,
   
When I look at patches uploaded to jiras, from time to time I notice
  that
different revisions of the patch is uploaded with the same patch file
   name,
some time for quite a few times. It's confusing which is which.
   
I'd suggest that as a guideline, we do the following when uploading a
patch:
   
   - include a revision number in the patch file name.A
   - include a comment, stating that a new patch is uploaded,
 including
   the
   revision number of the patch in the comment.
   
This way, it's easier to refer to a specific version of a patch, and
 to
know which patch a comment is made about.
   
Hope that makes sense to you.
   
Thanks.
   
--Yongjun
   
  
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

[jira] [Created] (HADOOP-11320) Submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HADOOP-11320:
--

 Summary: Submitting a hadoop patch doesn't trigger jenkins test run
 Key: HADOOP-11320
 URL: https://issues.apache.org/jira/browse/HADOOP-11320
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Yongjun Zhang


See details in INFRA-8655.

Per [~abayer] and [~cnauroth]'s feedback there , I'm creating this jira to 
investigate the possible bug in dev-support/test-patch.sh script.

Thanks Andrew and Chris.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Thanks for your input Steve!

--Yongjun


On Wed, Nov 19, 2014 at 1:45 AM, Steve Loughran ste...@hortonworks.com
wrote:

 If you have a change that spans the projects you need to submit a separate
 JIRA for each one anyway, to give each team the ability to
 review/accept/reject the patch

 On 18 November 2014 23:58, Ray Chiang rchi...@cloudera.com wrote:

  I had to go back and look, but it was Akira Ajisaka that had recommended
 it
  back when I ran into the issue:
 
 
 
 https://issues.apache.org/jira/browse/HADOOP-10946?focusedCommentId=14108880page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14108880
 
  Fortunately, my patch was easy to split up.
 
  -Ray
 
 
  On Tue, Nov 18, 2014 at 2:24 PM, Chris Nauroth cnaur...@hortonworks.com
 
  wrote:
 
   That's an interesting find.  This has not always been the case.  In the
   past, we've provided consolidated patches on a HADOOP jira to get
   pre-commit coverage when changes span multiple sub-projects.  Yongjun,
 if
   you haven't already done so, I suggest filing a jira for this.
  
   Chris Nauroth
   Hortonworks
   http://hortonworks.com/
  
  
   On Sat, Nov 15, 2014 at 8:19 AM, Yongjun Zhang yzh...@cloudera.com
   wrote:
  
For all's info, confirmed what Ray stated, indeed it's the reason!
   
Thanks.
   
--Yongjun
   
   
On Fri, Nov 14, 2014 at 2:40 PM, Yongjun Zhang yzh...@cloudera.com
wrote:
   
 Many thanks Ray, good to know!

 --Yongjun

 On Fri, Nov 14, 2014 at 2:03 PM, Ray Chiang rchi...@cloudera.com
wrote:

 Your patch covers multiple projects (I see 81 files across
 hadoop-common-project, hadoop-hdfs-project,
  hadoop-mapreduce-project,
 hadoop-tools, hadoop-yarn-project).

 Jenkins had problems with that for me a few months back.  If you
  break
it
 up per-project, it should work just fine.

 -Ray


 On Fri, Nov 14, 2014 at 1:49 PM, Ted Yu yuzhih...@gmail.com
  wrote:

  Adding builds@apache
 
  On Fri, Nov 14, 2014 at 1:34 PM, Yongjun Zhang 
  yzh...@cloudera.com
   
  wrote:
 
   Hi,
  
   One issue to report here, any help would be greatly
 appreciated!
  
   I noticed that multiple patch submissions to
   https://issues.apache.org/jira/browse/HADOOP-11293
   did not trigger jenkins test run.
  
   Thanks Chris Nauroth for the help to trigger one manually for
  me:
  
   https://builds.apache.org/job/PreCommit-HADOOP-Build/5079/
  
   and it turned out the manually triggered one did not report
  result
 back
  to
   the jira upon finishing.
  
   I submitted the same patch to another jira (HADOOP-11045)
 that I
   was
   working on, which triggered jenkins test before, and it seems
  not
   triggering this time too.
  
   So it looks like something is broken. This might be just a
  HADOOP
jira
   issue (HDFS jira seems to be fine).
  
   Thanks a lot for looking into.
  
   --Yongjun
  
 



   
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Thanks Chris and Ray, I created
https://issues.apache.org/jira/browse/INFRA-8655 for this matter.


--Yongjun


On Tue, Nov 18, 2014 at 2:58 PM, Ray Chiang rchi...@cloudera.com wrote:

 I had to go back and look, but it was Akira Ajisaka that had recommended it
 back when I ran into the issue:


 https://issues.apache.org/jira/browse/HADOOP-10946?focusedCommentId=14108880page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14108880

 Fortunately, my patch was easy to split up.

 -Ray


 On Tue, Nov 18, 2014 at 2:24 PM, Chris Nauroth cnaur...@hortonworks.com
 wrote:

  That's an interesting find.  This has not always been the case.  In the
  past, we've provided consolidated patches on a HADOOP jira to get
  pre-commit coverage when changes span multiple sub-projects.  Yongjun, if
  you haven't already done so, I suggest filing a jira for this.
 
  Chris Nauroth
  Hortonworks
  http://hortonworks.com/
 
 
  On Sat, Nov 15, 2014 at 8:19 AM, Yongjun Zhang yzh...@cloudera.com
  wrote:
 
   For all's info, confirmed what Ray stated, indeed it's the reason!
  
   Thanks.
  
   --Yongjun
  
  
   On Fri, Nov 14, 2014 at 2:40 PM, Yongjun Zhang yzh...@cloudera.com
   wrote:
  
Many thanks Ray, good to know!
   
--Yongjun
   
On Fri, Nov 14, 2014 at 2:03 PM, Ray Chiang rchi...@cloudera.com
   wrote:
   
Your patch covers multiple projects (I see 81 files across
hadoop-common-project, hadoop-hdfs-project,
 hadoop-mapreduce-project,
hadoop-tools, hadoop-yarn-project).
   
Jenkins had problems with that for me a few months back.  If you
 break
   it
up per-project, it should work just fine.
   
-Ray
   
   
On Fri, Nov 14, 2014 at 1:49 PM, Ted Yu yuzhih...@gmail.com
 wrote:
   
 Adding builds@apache

 On Fri, Nov 14, 2014 at 1:34 PM, Yongjun Zhang 
 yzh...@cloudera.com
  
 wrote:

  Hi,
 
  One issue to report here, any help would be greatly appreciated!
 
  I noticed that multiple patch submissions to
  https://issues.apache.org/jira/browse/HADOOP-11293
  did not trigger jenkins test run.
 
  Thanks Chris Nauroth for the help to trigger one manually for
 me:
 
  https://builds.apache.org/job/PreCommit-HADOOP-Build/5079/
 
  and it turned out the manually triggered one did not report
 result
back
 to
  the jira upon finishing.
 
  I submitted the same patch to another jira (HADOOP-11045) that I
  was
  working on, which triggered jenkins test before, and it seems
 not
  triggering this time too.
 
  So it looks like something is broken. This might be just a
 HADOOP
   jira
  issue (HDFS jira seems to be fine).
 
  Thanks a lot for looking into.
 
  --Yongjun
 

   
   
   
  
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

Re: [VOTE] Release Apache Hadoop 2.6.0

[Yongjun Zhang]

Hi Suresh,

I did review HDFS-6833 and the patch Shinichi provided a while back. It's
not really directly related to replication factor. The problem reported
there is, DirectoryScanner tries to register with NN blocks already
scheduled to be deleted as regular blocks, due to the delay in block
removal at DN side. This is wrong, and caused extra cycle/bandwidth and it
should not happen. And the patch tries to fix that.

I think it's a good fix to have, but probably targetting at 2.6.1 would be
fine. Given that the fix is still pending a committer's review, it will
take some time and we should push on that. I do hope the fix can get in as
soon as possible though.

Would you please share with us the timeframe for 2.6.1, and I hope the
schedule will work with Tsuyoshi.

Thanks.

--Yongjun


On Mon, Nov 17, 2014 at 10:03 AM, Suresh Srinivas sur...@hortonworks.com
wrote:

 Tsuyoshi, thanks for bringing up HDFS-6833. However, given it is a boundary
 condition (and should not cause issues when for files with replication
 factor 3), we should perhaps target this into 2.6.1 and not block this
 release. Thoughts?

 On Mon, Nov 17, 2014 at 4:17 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
 wrote:

  +0(non-binding)
 
  HDFS-6833 is critical issue for us - could you help us to merge it into
  2.6?
 
  Thanks,
  - Tsuyoshi
 
  On Sat, Nov 15, 2014 at 1:03 PM, Hitesh Shah hit...@apache.org wrote:
   +1 (binding)
  
   Built Hadoop from source, compiled Tez against the hadoop jars pushed
 to
  staging repo and ran a few example Tez jobs on a single node cluster.
  
   — HItesh
  
  
   On Nov 13, 2014, at 3:08 PM, Arun C Murthy a...@hortonworks.com
 wrote:
  
   Folks,
  
   I've created another release candidate (rc1) for hadoop-2.6.0 based on
  the feedback.
  
   The RC is available at:
  http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
   The RC tag in git is: release-2.6.0-rc1
  
   The maven artifacts are available via repository.apache.org at
  https://repository.apache.org/content/repositories/orgapachehadoop-1013.
  
   Please try the release and vote; the vote will run for the usual 5
 days.
  
   thanks,
   Arun
  
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
  entity to
   which it is addressed and may contain information that is
 confidential,
   privileged and exempt from disclosure under applicable law. If the
  reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 
 
 
  --
  - Tsuyoshi
 



 --
 http://hortonworks.com/download/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

For all's info, confirmed what Ray stated, indeed it's the reason!

Thanks.

--Yongjun


On Fri, Nov 14, 2014 at 2:40 PM, Yongjun Zhang yzh...@cloudera.com wrote:

 Many thanks Ray, good to know!

 --Yongjun

 On Fri, Nov 14, 2014 at 2:03 PM, Ray Chiang rchi...@cloudera.com wrote:

 Your patch covers multiple projects (I see 81 files across
 hadoop-common-project, hadoop-hdfs-project, hadoop-mapreduce-project,
 hadoop-tools, hadoop-yarn-project).

 Jenkins had problems with that for me a few months back.  If you break it
 up per-project, it should work just fine.

 -Ray


 On Fri, Nov 14, 2014 at 1:49 PM, Ted Yu yuzhih...@gmail.com wrote:

  Adding builds@apache
 
  On Fri, Nov 14, 2014 at 1:34 PM, Yongjun Zhang yzh...@cloudera.com
  wrote:
 
   Hi,
  
   One issue to report here, any help would be greatly appreciated!
  
   I noticed that multiple patch submissions to
   https://issues.apache.org/jira/browse/HADOOP-11293
   did not trigger jenkins test run.
  
   Thanks Chris Nauroth for the help to trigger one manually for me:
  
   https://builds.apache.org/job/PreCommit-HADOOP-Build/5079/
  
   and it turned out the manually triggered one did not report result
 back
  to
   the jira upon finishing.
  
   I submitted the same patch to another jira (HADOOP-11045) that I was
   working on, which triggered jenkins test before, and it seems not
   triggering this time too.
  
   So it looks like something is broken. This might be just a HADOOP jira
   issue (HDFS jira seems to be fine).
  
   Thanks a lot for looking into.
  
   --Yongjun
  
 

Re: [VOTE] Release Apache Hadoop 2.6.0

[Yongjun Zhang]

Thanks Arun for leading the 2.6 release effort.

+1 (non-binding)

- Downloaded rc1 source and did build
- Created two single-node clusters running 2.6
- Ran sample mapreduce job
- Ran distcp between two clusters

--Yongjun


On Thu, Nov 13, 2014 at 3:08 PM, Arun C Murthy a...@hortonworks.com wrote:

 Folks,

 I've created another release candidate (rc1) for hadoop-2.6.0 based on the
 feedback.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
 The RC tag in git is: release-2.6.0-rc1

 The maven artifacts are available via repository.apache.org at
 https://repository.apache.org/content/repositories/orgapachehadoop-1013.

 Please try the release and vote; the vote will run for the usual 5 days.

 thanks,
 Arun


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

submitting a hadoop patch doesn't trigger jenkins test run

[Yongjun Zhang]

Hi,

One issue to report here, any help would be greatly appreciated!

I noticed that multiple patch submissions to
https://issues.apache.org/jira/browse/HADOOP-11293
did not trigger jenkins test run.

Thanks Chris Nauroth for the help to trigger one manually for me:

https://builds.apache.org/job/PreCommit-HADOOP-Build/5079/

and it turned out the manually triggered one did not report result back to
the jira upon finishing.

I submitted the same patch to another jira (HADOOP-11045) that I was
working on, which triggered jenkins test before, and it seems not
triggering this time too.

So it looks like something is broken. This might be just a HADOOP jira
issue (HDFS jira seems to be fine).

Thanks a lot for looking into.

--Yongjun