[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646773#comment-16646773
 ] 

Ted Yu commented on HBASE-21247:


I tried another way of initializing WALFactory :
{code}
final WALFactory walFactory = new WALFactory(config, 
this.currentServername.toString(), false);
{code}
The test still fails with same assertion error.

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 
> 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646724#comment-16646724
 ] 

Ted Yu commented on HBASE-21247:


[~busbey]:
Please take a look at the attached 21247.v4.tst 
Running against current implementation, you would see:
{code}
testCustomProvider(org.apache.hadoop.hbase.wal.TestWALFactory)  Time elapsed: 
0.414 sec  <<< FAILURE!
java.lang.AssertionError: expected: but was:
at 
org.apache.hadoop.hbase.wal.TestWALFactory.testCustomProvider(TestWALFactory.java:740)
{code}

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 
> 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21247:
---
Attachment: 21247.v4.tst

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, 
> 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646681#comment-16646681
 ] 

Ted Yu commented on HBASE-21247:


Reverted to allow time for more discussion.

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20140) HRegion FileSystem should be instantiated from hbase rootDir not default

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646647#comment-16646647
 ] 

Ted Yu commented on HBASE-20140:


Is it possible to add a unit test ?

> HRegion FileSystem should be instantiated from hbase rootDir not default
> 
>
> Key: HBASE-20140
> URL: https://issues.apache.org/jira/browse/HBASE-20140
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 3.0.0, 2.0.2, 1.4.7
>Reporter: Adrian Muraru
>Assignee: Adrian Muraru
>Priority: Minor
> Attachments: HBASE-20140.branch-1.v01.patch, 
> HBASE-20140.branch-2.v01.patch, HBASE-20140.v01.patch
>
>
> HRegion fs initialization is done based on HDFS default {{fs.defaultFs}} 
> instead of {{hbase.rootdir}}
> This is breaking in cases where the {{fs.defaultFs}} is set to a different  
> filesystem than the one set in fully qualified {{hbase.rootdir}}
>  
> The usecase is the following: running an yarn cluster with `fs.defaultFs` set 
> to local HDFS and running a MR job over HBase Snapshots setting the 
> {{hbase.rootDir}} to external S3 Filesystem (fully qualified). In this case 
> when the regions are actually cold loaded in map tasks the defaultFs is used 
> instead of actual hbase.rootDir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646466#comment-16646466
 ] 

Ted Yu commented on HBASE-21247:


We are aware of the above (existing) config.
However, it is an enum. Meaning, hbase has to know the implementation.

This issue adds the ability for third party to plug in WAL provider without 
hbase hardcoding the implementation.

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21292) IdLock.getLockEntry() may hang if interrupted

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646408#comment-16646408
 ] 

Ted Yu commented on HBASE-21292:


Good catch.
For the second hunk, would it be more readable if the handling is moved to 
finally block ?

> IdLock.getLockEntry() may hang if interrupted
> -
>
> Key: HBASE-21292
> URL: https://issues.apache.org/jira/browse/HBASE-21292
> Project: HBase
>  Issue Type: Bug
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.1.0, 2.0.2, 1.4.9
>
> Attachments: HBASE-21292.branch-2.0.001.patch
>
>
> This is a rare case found by my colleague which really happened on our 
> production env. 
> Thread may hang(or enter a infinite loop ) when try to call 
> IdLock.getLockEntry(). Here is the case:
> 1. Thread1 owned the IdLock, while Thread2(the only one waiting) was waiting 
> for it.
> 2. Thread1 called releaseLockEntry, it will set IdLock.locked = false, but 
> since Thread2 was waiting, it won't call map.remove(entry.id)
> 3. While Thread1 was calling releaseLockEntry, Thread2 was interrupted. So no 
> one will remove this IdLock from the map.
> 4. If another thread try to call getLockEntry on this IdLock， it will end up 
> in a infinite loop. Since existing = map.putIfAbsent(entry.id, entry)) != 
> null and existing.locked=false
> It is hard to write a UT since it is a very rare race condition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20140) HRegion FileSystem should be instantiated from hbase rootDir not default

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646397#comment-16646397
 ] 

Ted Yu commented on HBASE-20140:


Can you get another qa run ?

The previous result was not accessible. 

> HRegion FileSystem should be instantiated from hbase rootDir not default
> 
>
> Key: HBASE-20140
> URL: https://issues.apache.org/jira/browse/HBASE-20140
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 3.0.0, 2.0.2, 1.4.7
>Reporter: Adrian Muraru
>Assignee: Adrian Muraru
>Priority: Minor
> Attachments: HBASE-20140.branch-1.v01.patch, 
> HBASE-20140.branch-2.v01.patch, HBASE-20140.v01.patch
>
>
> HRegion fs initialization is done based on HDFS default {{fs.defaultFs}} 
> instead of {{hbase.rootdir}}
> This is breaking in cases where the {{fs.defaultFs}} is set to a different  
> filesystem than the one set in fully qualified {{hbase.rootdir}}
>  
> The usecase is the following: running an yarn cluster with `fs.defaultFs` set 
> to local HDFS and running a MR job over HBase Snapshots setting the 
> {{hbase.rootDir}} to external S3 Filesystem (fully qualified). In this case 
> when the regions are actually cold loaded in map tasks the defaultFs is used 
> instead of actual hbase.rootDir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21281) Update bouncycastle dependency.

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645747#comment-16645747
 ] 

Ted Yu commented on HBASE-21281:


lgtm

> Update bouncycastle dependency.
> ---
>
> Key: HBASE-21281
> URL: https://issues.apache.org/jira/browse/HBASE-21281
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2
>
> Attachments: HBASE-21281.001.branch-2.0.patch
>
>
> Looks like we still depend on bcprov-jdk16 for some x509 certificate 
> generation in our tests. Bouncycastle has moved beyond this in 1.47, changing 
> the artifact names.
> [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later]
> There are some API changes too, but it looks like we don't use any of these.
> It seems like we also have vestiges in the POMs from when we were depending 
> on a specific BC version that came in from Hadoop. We now have a 
> KeyStoreTestUtil class in HBase, which makes me think we can also clean up 
> some dependencies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21246:
---
Attachment: 21246.HBASE-20952.005.patch

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, 
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch, 
> 21246.HBASE-20952.005.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21247:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the review, Josh.

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645443#comment-16645443
 ] 

Ted Yu commented on HBASE-21246:


I ran TestMasterFailoverWithProcedures with patch locally which passed.

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, 
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21246:
---
Attachment: 21246.HBASE-20952.004.patch

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, 
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645355#comment-16645355
 ] 

Ted Yu commented on HBASE-21246:


bq. Why isn't this a WALProvider method?

The start time of a WAL is known at the creation of the WAL. This attribute 
would not change even after edits are appended to the WAL.
Therefore I think it belongs to the identity.

bq. Seems unused.

This would be used in the (Ankit's) upcoming patch for replication.

The other comments are addressed by patch v3.

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, 
> 21246.HBASE-20952.002.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21246:
---
Attachment: 21246.003.patch

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, 
> 21246.HBASE-20952.002.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21246:
---
Attachment: 21246.HBASE-20952.002.patch

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638994#comment-16638994
 ] 

Ted Yu edited comment on HBASE-21246 at 10/10/18 3:56 PM:
--

The first question is addressed by patch v2 which drops the method.

The abstraction of WAL identity should allow all backends (hdfs, Ratis, Kafka) 
to express the information the underlying backend carries for each WAL.
e.g. for Kafka backend, we can use topic to represent region, the partitions 
under the topic would correspond to the WALs for the region.
Therefore a String alone doesn't have as much expressibility as an interface.

That is why we chose to use an interface to represent WAL identity.


was (Author: yuzhih...@gmail.com):
Now that design doc is back to review mode, please allow some more time till I 
address the above comment.

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21286) Parallelize computeHDFSBlocksDistribution when getting splits of a HBaseSnapshot

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645162#comment-16645162
 ] 

Ted Yu commented on HBASE-21286:


Where is the ExecutorService shutdown ?
{code}
432 ExecutorService executor = Executors.newFixedThreadPool(nrThreads);
{code}
If you can share some data on performance improvement, that would be nice.

> Parallelize computeHDFSBlocksDistribution when getting splits of a 
> HBaseSnapshot
> 
>
> Key: HBASE-21286
> URL: https://issues.apache.org/jira/browse/HBASE-21286
> Project: HBase
>  Issue Type: Improvement
>  Components: snapshots
>Affects Versions: 1.4.0
>Reporter: Lavinia-Stefania Sirbu
>Priority: Minor
> Attachments: HBASE-21286.branch-1.4.001.patch
>
>
> Even if this step is called computeHDFSBlocksDistribution, this is executed 
> no matter the file system of the snapshot. For example, we have observed an 
> important slowness when we have a snapshot in s3 (~26k regions, 5column 
> families, 2 files per column family) the getsplits time is ~40min due to the 
> calls in s3 for listing the files to get the best locations.
> Parallelizing this operation can reduce the overall setup time. The thread 
> pool should be configurable and a good choice could be 
> "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645138#comment-16645138
 ] 

Ted Yu commented on HBASE-21285:


In TableSnapshotInputFormatImpl#getRegionSizes, snapshot visitor may encounter 
IOExceptiion.
{code}
+Map regionSizes = getRegionSizes(conf, fs, snapshotDir);
{code}
Please add protection against potential IOExceptiion.

As I mentioned above, if the additional computation takes non-trivial amount of 
time, we should consider introducing new config for the improvement.

> Enhanced TableSnapshotInputFormat to allow a size based splitting
> -
>
> Key: HBASE-21285
> URL: https://issues.apache.org/jira/browse/HBASE-21285
> Project: HBase
>  Issue Type: Improvement
>  Components: snapshots
>Affects Versions: 1.4.0
>Reporter: Lavinia-Stefania Sirbu
>Priority: Minor
> Attachments: HBASE-21285.branch-1.4.001.patch
>
>
> Currently, all the splits generated by a snapshot are having length 0. Right 
> now, we have a configuration for the number of splits per region, but it's a 
> general one and not very helpful when the sizes for regions are really 
> different. The modification must be done in TableSnapshotInputFormatImpl 
> where the length must be computed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645099#comment-16645099
 ] 

Ted Yu commented on HBASE-21247:


If there is no further review comment, I plan to integrate this.

> Allow WAL Provider to be specified by configuration without explicit enum in 
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in 
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by 
> class name.
> This issue introduces additional config which allows the specification of new 
> WAL Provider through class name.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644985#comment-16644985
 ] 

Ted Yu commented on HBASE-21285:


Have you measured the duration for the {{getRegionSizes}} method on the 
snapshot(s) ?

Just wondering how much time would be spent on the extra computation.

Thanks

> Enhanced TableSnapshotInputFormat to allow a size based splitting
> -
>
> Key: HBASE-21285
> URL: https://issues.apache.org/jira/browse/HBASE-21285
> Project: HBase
>  Issue Type: Improvement
>  Components: snapshots
>Affects Versions: 1.4.0
>Reporter: Lavinia-Stefania Sirbu
>Priority: Minor
> Attachments: HBASE-21285.branch-1.4.001.patch
>
>
> Currently, all the splits generated by a snapshot are having length 0. Right 
> now, we have a configuration for the number of splits per region, but it's a 
> general one and not very helpful when the sizes for regions are really 
> different. The modification must be done in TableSnapshotInputFormatImpl 
> where the length must be computed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Apache Ratis Thirdparty 0.1.0rc0

[Ted Yu]

+1

On Mon, Oct 8, 2018 at 4:29 PM Josh Elser  wrote:

> Hi,
>
> Please vote on the following release candidate to become Apache Ratis
> Thirdparty 0.1.0.
>
> The Apache Ratis Thirdparty project is a collection of all thirdparty
> dependencies that Apache Ratis uses, repackaged for optimal use by
> Ratis. As such, there is very little net-new source code in this project.
>
> The source release is present at
>
> https://dist.apache.org/repos/dist/dev/incubator/ratis/thirdparty/0.1.0-rc0/
>
> SHA512 checksum is on the source tarball: A8A11CD5 447CD3AB 2341C878
> E3DB4AEF 565224E5 FEC599DF 36143EAB 0999437B 767860FE
>   54E3DC2C D4A265D1 E844979E 4D311CDD 97BC7797 CF66219A 3331D9F1
>
> This source release was created from the Git commit SHA1:
> 9e525162fcec8650b2d08dfdcbcdaee76231cc08. For your convenience, there is
> also a GPG-signed tag with the name "ratis-thirdparty-0.1.0" that also
> points at this commit.
>
> This source release was signed with my key: 4677D66C. This is present in
> the KEYS file (dist/dev and dist/release).
>
> The corresponding "binaries" for this release are staged at
> https://repository.apache.org/content/repositories/orgapacheratis-1007/
> and will be promoted pending successful PPMC and IPMC votes. You can
> update your local ~/.m2/settings.xml to add this as a repository to test
> the build of ratis.git if you choose (appears to work fine for me).
>
> This vote will be open for at least 72hours (until 2018/10/12 00:00:00
> GMT).
>
> - Josh
>
> ps: big thanks to Nicholas and Marton who helped out with this release
> (sorry if I'm forgetting anyone else!)
>

[jira] [Commented] (HBASE-21283) Add new shell command 'rit' for listing regions in transition

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644244#comment-16644244
 ] 

Ted Yu commented on HBASE-21283:


+1

> Add new shell command 'rit' for listing regions in transition
> -
>
> Key: HBASE-21283
> URL: https://issues.apache.org/jira/browse/HBASE-21283
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21283-branch-1.patch, HBASE-21283-branch-1.patch, 
> HBASE-21283.patch, HBASE-21283.patch
>
>
> The 'status' shell command shows regions in transition but sometimes an 
> operator may want to retrieve a simple list of regions in transition. Here's 
> a patch that adds a new 'rit' command to the TOOLS group that does just that. 
> No test, because it seems hard to mock RITs from the ruby test code, but I 
> have run TestShell and it passes, so the command is verified to meet minimum 
> requirements, like help text, and manually verified with branch-1 (shell in 
> branch-2 and up doesn't return until TransitRegionProcedure has completed so 
> by that time no RIT):
> {noformat}
> HBase Shell
> Use "help" to get list of supported commands.
> Use "exit" to quit this interactive shell.
> Version 1.5.0-SNAPSHOT, r9bb6d2fa8b760f16cd046657240ebd4ad91cb6de, Mon Oct  8 
> 21:05:50 UTC 2018
> hbase(main):001:0> help 'rit'
> List all regions in transition.
> Examples:
>   hbase> rit
> hbase(main):002:0> create ...
> 0 row(s) in 2.5150 seconds
> => Hbase::Table - IntegrationTestBigLinkedList
> hbase(main):003:0> rit
> 0 row(s) in 0.0340 seconds
> hbase(main):004:0> unassign '56f0c38c81ae453d19906ce156a2d6a1'
> 0 row(s) in 0.0540 seconds
> hbase(main):005:0> rit 
> IntegrationTestBigLinkedList,L\xCC\xCC\xCC\xCC\xCC\xCC\xCB,1539117183224.56f0c38c81ae453d19906ce156a2d6a1.
>  state=PENDING_CLOSE, ts=Tue Oct 09 20:33:34 UTC 2018 (0s ago), server=null   
>   
>   
> 
> 1 row(s) in 0.0170 seconds
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21283) Add new shell command 'rit' for listing regions in transition

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644225#comment-16644225
 ] 

Ted Yu commented on HBASE-21283:


{{getRegionStatesInTransition}} returns {{List}}
Would the multiple region states be concatenated with newline(s) in shell 
output ?


> Add new shell command 'rit' for listing regions in transition
> -
>
> Key: HBASE-21283
> URL: https://issues.apache.org/jira/browse/HBASE-21283
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21283-branch-1.patch, HBASE-21283-branch-1.patch, 
> HBASE-21283.patch, HBASE-21283.patch
>
>
> The 'status' shell command shows regions in transition but sometimes an 
> operator may want to retrieve a simple list of regions in transition. Here's 
> a patch that adds a new 'rit' command to the TOOLS group that does just that. 
> No test, because it seems hard to mock RITs from the ruby test code, but I 
> have run TestShell and it passes, so the command is verified to meet minimum 
> requirements, like help text, and manually verified with branch-1 (shell in 
> branch-2 and up doesn't return until TransitRegionProcedure has completed so 
> by that time no RIT):
> {noformat}
> HBase Shell
> Use "help" to get list of supported commands.
> Use "exit" to quit this interactive shell.
> Version 1.5.0-SNAPSHOT, r9bb6d2fa8b760f16cd046657240ebd4ad91cb6de, Mon Oct  8 
> 21:05:50 UTC 2018
> hbase(main):001:0> help 'rit'
> List all regions in transition.
> Examples:
>   hbase> rit
> hbase(main):002:0> create ...
> 0 row(s) in 2.5150 seconds
> => Hbase::Table - IntegrationTestBigLinkedList
> hbase(main):003:0> rit
> 0 row(s) in 0.0340 seconds
> hbase(main):004:0> unassign '56f0c38c81ae453d19906ce156a2d6a1'
> 0 row(s) in 0.0540 seconds
> hbase(main):005:0> rit 
> IntegrationTestBigLinkedList,L\xCC\xCC\xCC\xCC\xCC\xCC\xCB,1539117183224.56f0c38c81ae453d19906ce156a2d6a1.
>  state=PENDING_CLOSE, ts=Tue Oct 09 20:33:34 UTC 2018 (0s ago), server=null   
>   
>   
> 
> 1 row(s) in 0.0170 seconds
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HADOOP-15831:

Attachment: HADOOP-15831.03.patch

> Include modificationTime in the toString method of CopyListingFileStatus
> 
>
> Key: HADOOP-15831
> URL: https://issues.apache.org/jira/browse/HADOOP-15831
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Attachments: HADOOP-15831.01.patch, HADOOP-15831.02.patch, 
> HADOOP-15831.03.patch
>
>
> I was looking at a DistCp error observed in hbase backup test:
> {code}
> 2018-10-08 18:12:03,067 WARN  [Thread-933] mapred.LocalJobRunner$Job(590): 
> job_local1175594345_0004
> java.io.IOException: Inconsistent sequence file: current chunk file 
> org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
>
> c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
>  length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
> org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
>
> 57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
>  length = 5142 aclEntries = null, xAttrs = null}
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
>   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> 2018-10-08 18:12:03,150 INFO  [Time-limited test] 
> mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask: 
> 1.0 mapProgress: 1.0
> {code}
> I noticed that modificationTime was not included in the toString of 
> CopyListingFileStatus.
> I propose including modificationTime so that it is easier to tell when the 
> respective files last change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HADOOP-15831:

Attachment: HADOOP-15831.02.patch

> Include modificationTime in the toString method of CopyListingFileStatus
> 
>
> Key: HADOOP-15831
> URL: https://issues.apache.org/jira/browse/HADOOP-15831
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Attachments: HADOOP-15831.01.patch, HADOOP-15831.02.patch
>
>
> I was looking at a DistCp error observed in hbase backup test:
> {code}
> 2018-10-08 18:12:03,067 WARN  [Thread-933] mapred.LocalJobRunner$Job(590): 
> job_local1175594345_0004
> java.io.IOException: Inconsistent sequence file: current chunk file 
> org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
>
> c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
>  length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
> org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
>
> 57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
>  length = 5142 aclEntries = null, xAttrs = null}
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
>   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> 2018-10-08 18:12:03,150 INFO  [Time-limited test] 
> mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask: 
> 1.0 mapProgress: 1.0
> {code}
> I noticed that modificationTime was not included in the toString of 
> CopyListingFileStatus.
> I propose including modificationTime so that it is easier to tell when the 
> respective files last change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HADOOP-15831:

Assignee: Ted Yu
  Status: Patch Available  (was: Open)

> Include modificationTime in the toString method of CopyListingFileStatus
> 
>
> Key: HADOOP-15831
> URL: https://issues.apache.org/jira/browse/HADOOP-15831
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Attachments: HADOOP-15831.01.patch
>
>
> I was looking at a DistCp error observed in hbase backup test:
> {code}
> 2018-10-08 18:12:03,067 WARN  [Thread-933] mapred.LocalJobRunner$Job(590): 
> job_local1175594345_0004
> java.io.IOException: Inconsistent sequence file: current chunk file 
> org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
>
> c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
>  length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
> org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
>
> 57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
>  length = 5142 aclEntries = null, xAttrs = null}
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
>   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> 2018-10-08 18:12:03,150 INFO  [Time-limited test] 
> mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask: 
> 1.0 mapProgress: 1.0
> {code}
> I noticed that modificationTime was not included in the toString of 
> CopyListingFileStatus.
> I propose including modificationTime so that it is easier to tell when the 
> respective files last change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HADOOP-15831:

Attachment: HADOOP-15831.01.patch

> Include modificationTime in the toString method of CopyListingFileStatus
> 
>
> Key: HADOOP-15831
> URL: https://issues.apache.org/jira/browse/HADOOP-15831
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Minor
> Attachments: HADOOP-15831.01.patch
>
>
> I was looking at a DistCp error observed in hbase backup test:
> {code}
> 2018-10-08 18:12:03,067 WARN  [Thread-933] mapred.LocalJobRunner$Job(590): 
> job_local1175594345_0004
> java.io.IOException: Inconsistent sequence file: current chunk file 
> org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
>
> c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
>  length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
> org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
>
> 57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
>  length = 5142 aclEntries = null, xAttrs = null}
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
>   at 
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
>   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> 2018-10-08 18:12:03,150 INFO  [Time-limited test] 
> mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask: 
> 1.0 mapProgress: 1.0
> {code}
> I noticed that modificationTime was not included in the toString of 
> CopyListingFileStatus.
> I propose including modificationTime so that it is easier to tell when the 
> respective files last change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Re: Does Hbase backup process support encryption while transporting the data from one cluster to other cluster

[Ted Yu]

bq. Does copyTable support hashing of data while copying?

No.

bq.  Same for distcp utility ?

The above would get better answer posting on hadoop mailing list.

Thanks

On Tue, Oct 9, 2018 at 5:28 AM neo0731  wrote:

>
> Question arises when migrating the data from one hbase table to another.
>
> Input
>
> To sync the production cluster data with dev cluster. Additionaly, while
> copying we need to re-hash the following fields: hashed_email, lexer_id,
> foo_imsi, foo_msn, signal_uid, bar_imsi.
>
> Question is : Does copyTable support hashing of data while copying? Same
> for
> distcp utility ? Is it possible to supply some example code in scala as
> well
>
> Any help on it would be much appreciated?
>
>
>
> --
> Sent from:
> http://apache-hbase.679495.n3.nabble.com/HBase-Developer-f679493.html
>

Re: [VOTE] Apache Ratis Thirdparty 0.1.0rc0

[Ted Yu]

True.

Though I wouldn't expect the 'mvn install' to fail :-)

On Mon, Oct 8, 2018 at 8:14 PM Sergey Soldatov 
wrote:

> I have the same result with mvn install. Actually, mvn clean install works
> fine.
>
> Thanks,
> Sergey
>
> On Mon, Oct 8, 2018 at 7:53 PM Ted Yu  wrote:
>
> > When running 'mvn install', I got:
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (default) on
> > project ratis-thirdparty: Error creating shaded jar: duplicate entry:
> > META-INF/services/org.apache.ratis.thirdparty.io.grpc.ServerProvider ->
> > [Help 1]
> > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> > goal org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (default) on
> > project ratis-thirdparty: Error creating shaded jar: duplicate entry:
> > META-INF/services/org.apache.ratis.thirdparty.io.grpc.ServerProvider
> >
> > Here is my environment:
> >
> > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> > MaxPermSize=812M; support was removed in 8.0
> > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> > 2018-06-17T11:33:14-07:00)
> > Maven home: /Users/tyu/apache-maven-3.5.4
> > Java version: 1.8.0_151, vendor: Oracle Corporation, runtime:
> > /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac"
> >
> > Did I use the correct command ?
> >
> > Thanks
> >
> > On Mon, Oct 8, 2018 at 4:29 PM Josh Elser  wrote:
> >
> > > Hi,
> > >
> > > Please vote on the following release candidate to become Apache Ratis
> > > Thirdparty 0.1.0.
> > >
> > > The Apache Ratis Thirdparty project is a collection of all thirdparty
> > > dependencies that Apache Ratis uses, repackaged for optimal use by
> > > Ratis. As such, there is very little net-new source code in this
> project.
> > >
> > > The source release is present at
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/ratis/thirdparty/0.1.0-rc0/
> > >
> > > SHA512 checksum is on the source tarball: A8A11CD5 447CD3AB 2341C878
> > > E3DB4AEF 565224E5 FEC599DF 36143EAB 0999437B 767860FE
> > >   54E3DC2C D4A265D1 E844979E 4D311CDD 97BC7797 CF66219A 3331D9F1
> > >
> > > This source release was created from the Git commit SHA1:
> > > 9e525162fcec8650b2d08dfdcbcdaee76231cc08. For your convenience, there
> is
> > > also a GPG-signed tag with the name "ratis-thirdparty-0.1.0" that also
> > > points at this commit.
> > >
> > > This source release was signed with my key: 4677D66C. This is present
> in
> > > the KEYS file (dist/dev and dist/release).
> > >
> > > The corresponding "binaries" for this release are staged at
> > >
> https://repository.apache.org/content/repositories/orgapacheratis-1007/
> > > and will be promoted pending successful PPMC and IPMC votes. You can
> > > update your local ~/.m2/settings.xml to add this as a repository to
> test
> > > the build of ratis.git if you choose (appears to work fine for me).
> > >
> > > This vote will be open for at least 72hours (until 2018/10/12 00:00:00
> > > GMT).
> > >
> > > - Josh
> > >
> > > ps: big thanks to Nicholas and Marton who helped out with this release
> > > (sorry if I'm forgetting anyone else!)
> > >
> >
>

Re: [VOTE] Apache Ratis Thirdparty 0.1.0rc0

[Ted Yu]

When running 'mvn install', I got:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (default) on
project ratis-thirdparty: Error creating shaded jar: duplicate entry:
META-INF/services/org.apache.ratis.thirdparty.io.grpc.ServerProvider ->
[Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (default) on
project ratis-thirdparty: Error creating shaded jar: duplicate entry:
META-INF/services/org.apache.ratis.thirdparty.io.grpc.ServerProvider

Here is my environment:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
MaxPermSize=812M; support was removed in 8.0
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
2018-06-17T11:33:14-07:00)
Maven home: /Users/tyu/apache-maven-3.5.4
Java version: 1.8.0_151, vendor: Oracle Corporation, runtime:
/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.6", arch: "x86_64", family: "mac"

Did I use the correct command ?

Thanks

On Mon, Oct 8, 2018 at 4:29 PM Josh Elser  wrote:

> Hi,
>
> Please vote on the following release candidate to become Apache Ratis
> Thirdparty 0.1.0.
>
> The Apache Ratis Thirdparty project is a collection of all thirdparty
> dependencies that Apache Ratis uses, repackaged for optimal use by
> Ratis. As such, there is very little net-new source code in this project.
>
> The source release is present at
>
> https://dist.apache.org/repos/dist/dev/incubator/ratis/thirdparty/0.1.0-rc0/
>
> SHA512 checksum is on the source tarball: A8A11CD5 447CD3AB 2341C878
> E3DB4AEF 565224E5 FEC599DF 36143EAB 0999437B 767860FE
>   54E3DC2C D4A265D1 E844979E 4D311CDD 97BC7797 CF66219A 3331D9F1
>
> This source release was created from the Git commit SHA1:
> 9e525162fcec8650b2d08dfdcbcdaee76231cc08. For your convenience, there is
> also a GPG-signed tag with the name "ratis-thirdparty-0.1.0" that also
> points at this commit.
>
> This source release was signed with my key: 4677D66C. This is present in
> the KEYS file (dist/dev and dist/release).
>
> The corresponding "binaries" for this release are staged at
> https://repository.apache.org/content/repositories/orgapacheratis-1007/
> and will be promoted pending successful PPMC and IPMC votes. You can
> update your local ~/.m2/settings.xml to add this as a repository to test
> the build of ratis.git if you choose (appears to work fine for me).
>
> This vote will be open for at least 72hours (until 2018/10/12 00:00:00
> GMT).
>
> - Josh
>
> ps: big thanks to Nicholas and Marton who helped out with this release
> (sorry if I'm forgetting anyone else!)
>

[jira] [Commented] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642620#comment-16642620
 ] 

Ted Yu commented on HBASE-21149:


It seems the Warning was due to check during concatenation phase of 
CopyCommitter.
The concatenation would happen when input file listing is present.

I added debug logging at the beginning of 
MapReduceBackupCopyJob$BackupDistCp#createInputFileListing which didn't show up 
in test output.

> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
>  Issue Type: Test
>  Components: backuprestore
>    Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch, 
> testIncrementalBackupWithBulkLoad-output.txt
>
>
> From 
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
>  :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test] 
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
>   at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
>   at 
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642469#comment-16642469
 ] 

Ted Yu commented on HBASE-21149:


See the attached test output above.

> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
>  Issue Type: Test
>  Components: backuprestore
>    Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch, 
> testIncrementalBackupWithBulkLoad-output.txt
>
>
> From 
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
>  :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test] 
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
>   at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
>   at 
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus

[Ted Yu (JIRA)]

Ted Yu created HADOOP-15831:
---

 Summary: Include modificationTime in the toString method of 
CopyListingFileStatus
 Key: HADOOP-15831
 URL: https://issues.apache.org/jira/browse/HADOOP-15831
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ted Yu


I was looking at a DistCp error observed in hbase backup test:
{code}
2018-10-08 18:12:03,067 WARN  [Thread-933] mapred.LocalJobRunner$Job(590): 
job_local1175594345_0004
java.io.IOException: Inconsistent sequence file: current chunk file 
org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
   
c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
 length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
   
57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
 length = 5142 aclEntries = null, xAttrs = null}
  at 
org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
  at 
org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
  at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
2018-10-08 18:12:03,150 INFO  [Time-limited test] 
mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask: 
1.0 mapProgress: 1.0
{code}
I noticed that modificationTime was not included in the toString of 
CopyListingFileStatus.

I propose including modificationTime so that it is easier to tell when the 
respective files last change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus

[Ted Yu (JIRA)]

Ted Yu created HADOOP-15831:
---

 Summary: Include modificationTime in the toString method of 
CopyListingFileStatus
 Key: HADOOP-15831
 URL: https://issues.apache.org/jira/browse/HADOOP-15831
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ted Yu


I was looking at a DistCp error observed in hbase backup test:
{code}
2018-10-08 18:12:03,067 WARN  [Thread-933] mapred.LocalJobRunner$Job(590): 
job_local1175594345_0004
java.io.IOException: Inconsistent sequence file: current chunk file 
org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
   
c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
 length = 5100 aclEntries  = null, xAttrs = null} doesnt match prior entry 
org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
   
57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
 length = 5142 aclEntries = null, xAttrs = null}
  at 
org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
  at 
org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
  at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
2018-10-08 18:12:03,150 INFO  [Time-limited test] 
mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask: 
1.0 mapProgress: 1.0
{code}
I noticed that modificationTime was not included in the toString of 
CopyListingFileStatus.

I propose including modificationTime so that it is easier to tell when the 
respective files last change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Commented] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642312#comment-16642312
 ] 

Ted Yu commented on HBASE-21149:


I reproduced test error running against hadoop 3.1.1 :
{code}
[ERROR] 
TestIncBackupDeleteTable(org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad)
  Time elapsed: 105.11 s  <<< ERROR!
java.io.IOException: java.io.IOException: Failed copy from 
hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_,hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
 to hdfs://localhost:41712/backupUT/backup_1539022312079
  at 
org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
Caused by: java.io.IOException: Failed copy from 
hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_,hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
 to hdfs://localhost:41712/backupUT/backup_1539022312079
  at 
org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
{code}

> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
>  Issue Type: Test
>  Components: backuprestore
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch, 
> testIncrementalBackupWithBulkLoad-output.txt
>
>
> From 
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
>  :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test] 
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
>   at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
>   at 
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
&g

[jira] [Updated] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21149:
---
Attachment: testIncrementalBackupWithBulkLoad-output.txt

> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
>  Issue Type: Test
>  Components: backuprestore
>    Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch, 
> testIncrementalBackupWithBulkLoad-output.txt
>
>
> From 
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
>  :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test] 
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from 
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
>  to hdfs://localhost:53075/backupUT/backup_1535975655488
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
>   at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
>   at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
>   at 
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21230) BackupUtils#checkTargetDir doesn't compose error message correctly

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21230:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the patch, liubang.

> BackupUtils#checkTargetDir doesn't compose error message correctly
> --
>
> Key: HBASE-21230
> URL: https://issues.apache.org/jira/browse/HBASE-21230
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>    Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21230-1.patch
>
>
> Here is related code from BackupUtils#checkTargetDir:
> {code}
>   String expMsg = e.getMessage();
>   String newMsg = null;
>   if (expMsg.contains("No FileSystem for scheme")) {
> newMsg =
> "Unsupported filesystem scheme found in the backup target url. 
> Error Message: "
> + newMsg;
> {code}
> I think the intention was to concatenate expMsg at the end of newMsg.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21279) Split TestAdminShell into several tests

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21279:
---
Attachment: testAdminShell-output.tar.gz

> Split TestAdminShell into several tests
> ---
>
> Key: HBASE-21279
> URL: https://issues.apache.org/jira/browse/HBASE-21279
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>Priority: Major
> Attachments: testAdminShell-output.tar.gz
>
>
> In the flaky test board, TestAdminShell often timed out 
> (https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).
> I ran the test on Linux with SSD and reproduced the timeout (see attached 
> test output).
> {code}
> 2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting 
> hbase.rootdir to 
> /mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
> ...
> 2018-10-08 02:49:09,093 DEBUG 
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7] 
> master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
> Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1] 
> util.FSTableDescriptors(684): Wrote into 
> hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-   
> d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
> 2018-10-08 02:49:09,328 INFO  
> [RegionOpenAndInitThread-hbase_shell_tests_table-1] 
> regionserver.HRegion(7004): creating HRegion hbase_shell_tests_table HTD ==   
>   'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5', 
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', 
> KEEP_DELETED_CELLS => 'FALSE',   CACHE_DATA_ON_WRITE => 
> 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => 
> '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 
> 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', 
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 
> 'true', BLOCKSIZE => '65536'},  {NAME => 'y', VERSIONS => '1', 
> EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', 
> KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',  
> DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', 
> REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 
> 'false', IN_MEMORY => 'false',  CACHE_BLOOMS_ON_WRITE => 'false', 
> PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 
> 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
> user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name == 
> hbase_shell_tests_table
> ^[[38;5;226mE^[[0m
> ===
> Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m: 
> Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation: 
> CREATE, Table Name:  default:hbase_shell_tests_table, procId: 871
> 2018-10-08 02:49:09,361 INFO  [Block report processor] 
> blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 
> 127.0.0.1:41338 is added to   
> blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null, 
> primaryNodeIndex=-1, 
> replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
>41338|RBW]]} size 58
> > TEST TIMED OUT. PRINTING THREAD DUMP. <
> {code}
> We can see that the procedure #871 wasn't stuck - the timeout cut in and 
> stopped the test.
> We should separate the current test into two (or more) test files (with 
> corresponding .rb) so that the execution time consistently would not exceed 
> limit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21279) Split TestAdminShell into several tests

[Ted Yu (JIRA)]

Ted Yu created HBASE-21279:
--

 Summary: Split TestAdminShell into several tests
 Key: HBASE-21279
 URL: https://issues.apache.org/jira/browse/HBASE-21279
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu


In the flaky test board, TestAdminShell often timed out 
(https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).

I ran the test on Linux with SSD and reproduced the timeout (see attached test 
output).
{code}
2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting 
hbase.rootdir to 
/mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
...
2018-10-08 02:49:09,093 DEBUG 
[RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7] 
master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1] 
util.FSTableDescriptors(684): Wrote into 
hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-   
d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
2018-10-08 02:49:09,328 INFO  
[RegionOpenAndInitThread-hbase_shell_tests_table-1] regionserver.HRegion(7004): 
creating HRegion hbase_shell_tests_table HTD == 
'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5', EVICT_BLOCKS_ON_CLOSE 
=> 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', 
  CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', 
TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 
'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', 
CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', 
COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'},  {NAME 
=> 'y', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR 
=> 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',  
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', 
REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 
'false', IN_MEMORY => 'false',  CACHE_BLOOMS_ON_WRITE => 'false', 
PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 
'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name == 
hbase_shell_tests_table
^[[38;5;226mE^[[0m
===
Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m: 
Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation: 
CREATE, Table Name:  default:hbase_shell_tests_table, procId: 871
2018-10-08 02:49:09,361 INFO  [Block report processor] 
blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 
127.0.0.1:41338 is added to   
blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
   41338|RBW]]} size 58
> TEST TIMED OUT. PRINTING THREAD DUMP. <
{code}
We can see that the procedure #871 wasn't stuck - the timeout cut in and 
stopped the test.

We should separate the current test into two (or more) test files (with 
corresponding .rb) so that the execution time consistently would not exceed 
limit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21279) Split TestAdminShell into several tests

[Ted Yu (JIRA)]

Ted Yu created HBASE-21279:
--

 Summary: Split TestAdminShell into several tests
 Key: HBASE-21279
 URL: https://issues.apache.org/jira/browse/HBASE-21279
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu


In the flaky test board, TestAdminShell often timed out 
(https://builds.apache.org/job/HBASE-Find-Flaky-Tests/job/branch-2/lastSuccessfulBuild/artifact/dashboard.html).

I ran the test on Linux with SSD and reproduced the timeout (see attached test 
output).
{code}
2018-10-08 02:36:09,146 DEBUG [main] hbase.HBaseTestingUtility(351): Setting 
hbase.rootdir to 
/mnt/disk2/a/2-hbase/hbase-shell/target/test-data/a103d8e4-695c-a5a9-6690-1ef2580050f9
...
2018-10-08 02:49:09,093 DEBUG 
[RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=7] 
master.MasterRpcServices(1171): Checking to see if procedure is done pid=871
Took 0.7262 seconds2018-10-08 02:49:09,324 DEBUG [PEWorker-1] 
util.FSTableDescriptors(684): Wrote into 
hdfs://localhost:43859/user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-   
d935316c9241/.tmp/data/default/hbase_shell_tests_table/.tabledesc/.tableinfo.01
2018-10-08 02:49:09,328 INFO  
[RegionOpenAndInitThread-hbase_shell_tests_table-1] regionserver.HRegion(7004): 
creating HRegion hbase_shell_tests_table HTD == 
'hbase_shell_tests_table', {NAME => 'x', VERSIONS => '5', EVICT_BLOCKS_ON_CLOSE 
=> 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', 
  CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', 
TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 
'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', 
CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', 
COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'},  {NAME 
=> 'y', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR 
=> 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false',  
DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', 
REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 
'false', IN_MEMORY => 'false',  CACHE_BLOOMS_ON_WRITE => 'false', 
PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 
'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43859/
user/hbase/test-data/cefc73d9-cc37-d2a6-b92b-d935316c9241/.tmp Table name == 
hbase_shell_tests_table
^[[38;5;226mE^[[0m
===
Error: ^[[48;5;16;38;5;226;1mtest_Get_simple_status(Hbase::StatusTest)^[[0m: 
Java::JavaIo::InterruptedIOException: Interrupt while waiting on Operation: 
CREATE, Table Name:  default:hbase_shell_tests_table, procId: 871
2018-10-08 02:49:09,361 INFO  [Block report processor] 
blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 
127.0.0.1:41338 is added to   
blk_1073742193_1369{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-ecc89143-e0a5-4a1c-b552-120be2561334:NORMAL:127.0.0.1:
   41338|RBW]]} size 58
> TEST TIMED OUT. PRINTING THREAD DUMP. <
{code}
We can see that the procedure #871 wasn't stuck - the timeout cut in and 
stopped the test.

We should separate the current test into two (or more) test files (with 
corresponding .rb) so that the execution time consistently would not exceed 
limit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21216:
---
Description: 
>From 
>https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
{code}
java.lang.AssertionError: Archived hfiles [] and table hfiles 
[9ca09392705f425f9c916beedc10d63c] is missing snapshot 
file:6739a09747e54189a4112a6d8f37e894
at 
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
{code}
The file appeared in archive dir before hfile cleaners were run:
{code}
2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-archive/
2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|data/
2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|---default/
2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|--test/
2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-1237d57b63a7bdf067a930441a02514a/
2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|recovered.edits/
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---4.seqid
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-29e1700e09b51223ad2f5811105a4d51/
2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|fam/
2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---2c66a18f6c1a4074b84ffbb3245268c4
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---45bb396c6a5e49629e45a4d56f1e9b14
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---6739a09747e54189a4112a6d8f37e894
{code}
However, the archive dir became empty after hfile cleaners were run:
{code}
2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-archive/
2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-corrupt/
{code}
Leading to the assertion failure.

This test is one of the top flaky tests.

  was:
>From 
>https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
{code}
java.lang.AssertionError: Archived hfiles [] and table hfiles 
[9ca09392705f425f9c916beedc10d63c] is missing snapshot 
file:6739a09747e54189a4112a6d8f37e894
at 
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
{code}
The file appeared in archive dir before hfile cleaners were run:
{code}
2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-archive/
2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|data/
2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|---default/
2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|--test/
2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-1237d57b63a7bdf067a930441a02514a/
2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|recovered.edits/
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---4.seqid
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-29e1700e09b51223ad2f5811105a4d51/
2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|fam/
2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---2c66a18f6c1a4074b84ffbb3245268c4
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---45bb396c6a5e49629e45a4d56f1e9b14
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774): 
|---6739a09747e54189a4112a6d8f37e894
{code}
However, the archive dir became empty after hfile cleaners were run:
{code}
2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-archive/
2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771): 
|-corrupt/
{code}
Leading to the assertion failure.


> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>Assignee: Ted Yu
>   

[jira] [Updated] (HBASE-21230) BackupUtils#checkTargetDir doesn't compose error message correctly

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21230:
---
Description: 
Here is related code from BackupUtils#checkTargetDir:
{code}
  String expMsg = e.getMessage();
  String newMsg = null;
  if (expMsg.contains("No FileSystem for scheme")) {
newMsg =
"Unsupported filesystem scheme found in the backup target url. 
Error Message: "
+ newMsg;
{code}
I think the intention was to concatenate expMsg at the end of newMsg.

  was:
Here is related code:
{code}
  String expMsg = e.getMessage();
  String newMsg = null;
  if (expMsg.contains("No FileSystem for scheme")) {
newMsg =
"Unsupported filesystem scheme found in the backup target url. 
Error Message: "
+ newMsg;
{code}
I think the intention was to concatenate expMsg at the end of newMsg.


> BackupUtils#checkTargetDir doesn't compose error message correctly
> --
>
> Key: HBASE-21230
> URL: https://issues.apache.org/jira/browse/HBASE-21230
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Minor
>
> Here is related code from BackupUtils#checkTargetDir:
> {code}
>   String expMsg = e.getMessage();
>   String newMsg = null;
>   if (expMsg.contains("No FileSystem for scheme")) {
> newMsg =
> "Unsupported filesystem scheme found in the backup target url. 
> Error Message: "
> + newMsg;
> {code}
> I think the intention was to concatenate expMsg at the end of newMsg.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21238) MapReduceHFileSplitterJob#run shouldn't call System.exit

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21238:
---
Labels: mapreduce  (was: )

> MapReduceHFileSplitterJob#run shouldn't call System.exit
> 
>
> Key: HBASE-21238
> URL: https://issues.apache.org/jira/browse/HBASE-21238
> Project: HBase
>  Issue Type: Bug
>    Reporter: Ted Yu
>Priority: Minor
>  Labels: mapreduce
>
> {code}
> if (args.length < 2) {
>   usage("Wrong number of arguments: " + args.length);
>   System.exit(-1);
> {code}
> Correct way of handling error condition is through return value of run method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved FLINK-10468.

Resolution: Not A Bug

> Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
> --
>
> Key: FLINK-10468
> URL: https://issues.apache.org/jira/browse/FLINK-10468
> Project: Flink
>  Issue Type: Bug
>    Reporter: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> switch (strategy) {
> case PARTITION_CUSTOM:
>   extractedKeys = new Object[1];
> case FORWARD:
> {code}
> It seems a 'break' is missing prior to FORWARD case.
> {code}
> if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null) 
> {
>   throw new NullPointerException("Partitioner must not be null when the 
> ship strategy is set to custom partitioning.");
> }
> {code}
> Since the above check is for PARTITION_CUSTOM, it seems we can place the 
> check in the switch statement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21219) Hbase incremental backup fails with null pointer exception

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21219:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the patch, Vlad.

> Hbase incremental backup fails with null pointer exception
> --
>
> Key: HBASE-21219
> URL: https://issues.apache.org/jira/browse/HBASE-21219
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21219-v1.patch
>
>
> hbase backup create incremental hdfs:///bkpHbase_Test/bkpHbase_Test2 -t 
> bkpHbase_Test2 
> 2018-09-21 15:35:31,421 INFO [main] impl.TableBackupClient: Backup 
> backup_1537524313995 started at 1537524331419. 2018-09-21 15:35:31,454 INFO 
> [main] impl.IncrementalBackupManager: Execute roll log procedure for 
> incremental backup ... 2018-09-21 15:35:32,985 ERROR [main] 
> impl.TableBackupClient: Unexpected Exception : java.lang.NullPointerException 
> java.lang.NullPointerException at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347)
>  at 
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138)
>  at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171) 
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
> org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179) 
> 2018-09-21 15:35:32,989 ERROR [main] impl.TableBackupClient: 
> BackupId=backup_1537524313995,startts=1537524331419,failedts=1537524332989,failedphase=PREPARE_INCREMENTAL,failedmessage=null
>  2018-09-21 15:35:57,167 ERROR [main] impl.TableBackupClient: Backup 
> backup_1537524313995 failed. 
> Backup session finished. Status: FAILURE 2018-09-21 15:35:57,175 ERROR [main] 
> backup.BackupDriver: Error running 
> command-line tool java.io.IOException: java.lang.NullPointerException at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:281)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347)
>  at 
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138)
>  at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171) 
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
> org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179) 
> Caused by: java.lang.NullPointerException at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276)
>  ... 7 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-274) Read/Write-path of log stream state machine

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/RATIS-274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640849#comment-16640849
 ] 

Ted Yu commented on RATIS-274:
--

{code}
+  LogStream createLog(LogName name, LogServiceConfiguration config);
{code}
The single parameter createLog throws exception. Should the above method throw 
exception as well ?
{code}
+   * @return
*/
-  void removeRecordListener(LogName name, RecordListener listener);
+  boolean removeRecordListener(LogName name, RecordListener listener);
{code}
What does the return value represent ?

For LogServiceConfiguration#set, Map#put returns:
the previous value associated with key, or null if there was no mapping for key

It would be good to align the set with Map#put in terms of return value.
{code}
+try {
+  log.close();
+  close();
+} catch (IOException e) {
{code}
If log.close throws exception, wouldn't close() call be skipped ?



> Read/Write-path of log stream state machine
> ---
>
> Key: RATIS-274
> URL: https://issues.apache.org/jira/browse/RATIS-274
> Project: Ratis
>  Issue Type: Sub-task
>  Components: LogService
>Reporter: Josh Elser
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: RATIS-274-v1.patch
>
>
> Implement the ability to read/write data to a log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21219) Hbase incremental backup fails with null pointer exception

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640845#comment-16640845
 ] 

Ted Yu commented on HBASE-21219:


Ping [~vrodionov]
If you agree with my comment above, I can adjust the log level upon commit.

> Hbase incremental backup fails with null pointer exception
> --
>
> Key: HBASE-21219
> URL: https://issues.apache.org/jira/browse/HBASE-21219
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21219-v1.patch
>
>
> hbase backup create incremental hdfs:///bkpHbase_Test/bkpHbase_Test2 -t 
> bkpHbase_Test2 
> 2018-09-21 15:35:31,421 INFO [main] impl.TableBackupClient: Backup 
> backup_1537524313995 started at 1537524331419. 2018-09-21 15:35:31,454 INFO 
> [main] impl.IncrementalBackupManager: Execute roll log procedure for 
> incremental backup ... 2018-09-21 15:35:32,985 ERROR [main] 
> impl.TableBackupClient: Unexpected Exception : java.lang.NullPointerException 
> java.lang.NullPointerException at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347)
>  at 
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138)
>  at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171) 
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
> org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179) 
> 2018-09-21 15:35:32,989 ERROR [main] impl.TableBackupClient: 
> BackupId=backup_1537524313995,startts=1537524331419,failedts=1537524332989,failedphase=PREPARE_INCREMENTAL,failedmessage=null
>  2018-09-21 15:35:57,167 ERROR [main] impl.TableBackupClient: Backup 
> backup_1537524313995 failed. 
> Backup session finished. Status: FAILURE 2018-09-21 15:35:57,175 ERROR [main] 
> backup.BackupDriver: Error running 
> command-line tool java.io.IOException: java.lang.NullPointerException at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:281)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
>  at 
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347)
>  at 
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138)
>  at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171) 
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at 
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at 
> org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179) 
> Caused by: java.lang.NullPointerException at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103)
>  at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276)
>  ... 7 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21272) Re-add assertions for RS Group admin tests

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21272:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the review, Andrew.

> Re-add assertions for RS Group admin tests
> --
>
> Key: HBASE-21272
> URL: https://issues.apache.org/jira/browse/HBASE-21272
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: 21272.branch-1.01.patch
>
>
> The checked in version of HBASE-21258 for branch-1 didn't include assertions 
> for adding / removing RS group coprocessor hook calls.
> This issue is to add the assertions to corresponding tests in 
> TestRSGroupsAdmin1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21273) Move classes out of org.apache.spark namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640488#comment-16640488
 ] 

Ted Yu commented on HBASE-21273:


lgtm

> Move classes out of org.apache.spark namespace
> --
>
> Key: HBASE-21273
> URL: https://issues.apache.org/jira/browse/HBASE-21273
> Project: HBase
>  Issue Type: Task
>  Components: spark
>Affects Versions: 3.0.0
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21273.master.001.patch
>
>
> We currently have classes in the org.apache.spark space, I expect the Spark 
> PMC would be upset with us if we started releasing those. Let's see if we can 
> move them out.
> There's an ancient (2016) comment saying we need them there for access 
> restriction reasons, if that's the case then we'll have to work through that 
> issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Replacing EasyMock with Mockito in Kafka

[Ted Yu]

+1 to moving to Mockito

On Fri, Oct 5, 2018 at 12:11 PM Ron Dagostino  wrote:

> I have used Mockito and am a big fan -- I had never used EasyMock until
> recently.  The concept of record vs. replay mode in EasyMock really annoyed
> me -- I'm a fan of the "NO MODES" idea (
> https://en.wikipedia.org/wiki/Larry_Tesler), and when I encountered it in
> EasyMock in the context of Kafka development I just gritted my teeth and
> learned/used it as required.  I'm very pleased that Mockito is being
> seriously considered as a replacement.  Mockito  started as an EasyMock
> fork, I guess, and the project readily admits it is standing on the
> shoulders of giants in many respects, but I think one of the best things
> they ever did was jettison the concept of modes.
>
> +1 from me.
>
> Ron
>
> On Fri, Oct 5, 2018 at 10:00 AM Ismael Juma  wrote:
>
> > So far, it seems like people are in favour. If no objections are
> presented
> > in the next couple of days, we will go ahead with the change.
> >
> > Ismael
> >
> > On Sun, 30 Sep 2018, 20:32 Ismael Juma,  wrote:
> >
> > > Hi all,
> > >
> > > As described in KAFKA-7438
> > > , EasyMock's
> > > development has stagnated. This presents a number of issues:
> > >
> > > 1. Blocks us from running tests with newer Java versions, which is a
> > > frequent occurrence give the new Java release cadence. It is the main
> > > blocker in switching Jenkins from Java 10 to Java 11 at the moment.
> > > 2. Integration with newer testing libraries like JUnit 5 is slow to
> > appear
> > > (if it appears at all).
> > > 3. No API improvements. Mockito started as an EasyMock fork, but has
> > > continued to evolve and, in my opinion, it's more intuitive now.
> > >
> > > I think we should switch to Mockito for new tests and to incrementally
> > > migrate the existing ones as time allows. To make the proposal
> concrete,
> > I
> > > went ahead and converted all the tests in the `clients` module:
> > >
> > > https://github.com/apache/kafka/pull/5691
> > >
> > > I think the updated tests are nicely readable. I also removed PowerMock
> > > from the `clients` tests as we didn't really need it and its
> development
> > > has also stagnated a few months ago. I think we can easily remove
> > PowerMock
> > > elsewhere with the exception of `Connect` where we may need to keep it
> > for
> > > a while.
> > >
> > > Let me know your thoughts. Aside from the general future direction, I'd
> > > like to get the PR for KAFKA-7439 reviewed and merged soonish as merge
> > > conflicts will creep in quickly.
> > >
> > > Ismael
> > >
> >
>

[jira] [Updated] (HBASE-21272) Re-add assertions for RS Group admin tests

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21272:
---
Attachment: 21272.branch-1.01.patch

> Re-add assertions for RS Group admin tests
> --
>
> Key: HBASE-21272
> URL: https://issues.apache.org/jira/browse/HBASE-21272
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: 21272.branch-1.01.patch
>
>
> The checked in version of HBASE-21258 for branch-1 didn't include assertions 
> for adding / removing RS group coprocessor hook calls.
> This issue is to add the assertions to corresponding tests in 
> TestRSGroupsAdmin1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21272) Re-add assertions for RS Group admin tests

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21272:
---
Status: Patch Available  (was: Open)

> Re-add assertions for RS Group admin tests
> --
>
> Key: HBASE-21272
> URL: https://issues.apache.org/jira/browse/HBASE-21272
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 1.5.0
>
> Attachments: 21272.branch-1.01.patch
>
>
> The checked in version of HBASE-21258 for branch-1 didn't include assertions 
> for adding / removing RS group coprocessor hook calls.
> This issue is to add the assertions to corresponding tests in 
> TestRSGroupsAdmin1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21272) Re-add assertions for RS Group admin tests

[Ted Yu (JIRA)]

Ted Yu created HBASE-21272:
--

 Summary: Re-add assertions for RS Group admin tests
 Key: HBASE-21272
 URL: https://issues.apache.org/jira/browse/HBASE-21272
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 1.5.0


The checked in version of HBASE-21258 for branch-1 didn't include assertions 
for adding / removing RS group coprocessor hook calls.

This issue is to add the assertions to corresponding tests in TestRSGroupsAdmin1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21272) Re-add assertions for RS Group admin tests

[Ted Yu (JIRA)]

Ted Yu created HBASE-21272:
--

 Summary: Re-add assertions for RS Group admin tests
 Key: HBASE-21272
 URL: https://issues.apache.org/jira/browse/HBASE-21272
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 1.5.0


The checked in version of HBASE-21258 for branch-1 didn't include assertions 
for adding / removing RS group coprocessor hook calls.

This issue is to add the assertions to corresponding tests in TestRSGroupsAdmin1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21265) Split up TestRSGroups

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639225#comment-16639225
 ] 

Ted Yu commented on HBASE-21265:


{code}
public class IntegrationTestRSGroup extends TestRSGroupsBase {
{code}
Now that the RS Group tests are broken out of TestRSGroupsBase, should 
IntegrationTestRSGroup incur similar changes ?

Thanks

> Split up TestRSGroups
> -
>
> Key: HBASE-21265
> URL: https://issues.apache.org/jira/browse/HBASE-21265
> Project: HBase
>  Issue Type: Task
>  Components: rsgroup, test
>Affects Versions: 1.4.8
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>  Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9
>
> Attachments: HBASE-21265-branch-1.patch, HBASE-21265-branch-1.patch, 
> HBASE-21265-branch-2.patch, HBASE-21265-branch-2.patch, HBASE-21265.patch, 
> HBASE-21265.patch
>
>
> TestRSGroups is flaky. It is stable when run in isolation but when run as 
> part of the suite with concurrent executors it can fail. The current running 
> time of this unit on my dev box is ~240 seconds (4 minutes), which is far too 
> much time. This unit should be broken up 5 to 8 ways, grouped by 
> functionality under test. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638994#comment-16638994
 ] 

Ted Yu commented on HBASE-21246:


Now that design doc is back to review mode, please allow some more time till I 
address the above comment.

> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Attachments: 21246.HBASE-20952.001.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can 
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent 
> filename in distributed filesystem environment or, the name of the stream 
> when the WAL is backed by log stream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] The first HBase 1.4.8 release candidate (RC0) is available

[Ted Yu]

+1

 - verified checksums and signatures: good
 - basic checking on Web UI : good
 - ran test suite with : good

Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
2018-06-17T18:33:14Z)
Maven home: /apache-maven-3.5.4
Java version: 1.8.0_161, vendor: Oracle Corporation, runtime:
/mnt/disk2/a/jdk1.8.0_161/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64",
family: "unix"

 - Ran LTT with 1M rows : good

On Wed, Oct 3, 2018 at 9:34 AM Andrew Purtell  wrote:

> RC errata:
>
> TestRSGroups will pass in isolation but may fail when run as part of the
> suite. There have been several JIRAs filed against this unit like
> HBASE-19444, HBASE-19461, HBASE-20137, and mentioned on HBASE-21187. It's
> running time is far too long. I have filed HBASE-21265 to split it up. Test
> stabilization work would be a part of that. I don't think  this rises to
> the level of failing the RC vote because TestRSGroups will pass
> consistently, at least for me, when run in isolation. I do agree that
> without work to improve the test it doesn't offer the kind of functional
> assurance we'd like to derive from a unit test.
>
>
> On Tue, Oct 2, 2018 at 5:57 PM Andrew Purtell  wrote:
>
> > The first HBase 1.4.8 release candidate (RC0) is available for download
> at
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.8RC0/ and Maven
> > artifacts are available in the temporary repository
> > https://repository.apache.org/content/repositories/orgapachehbase-1233/
> >
> > The git tag corresponding to the candidate is '1.4.8RC0' (91118ce5f1).
> >
> > A detailed source and binary compatibility report for this release is
> > available for your review at
> >
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.4.8RC0/compat-check-report.html
> > . There are no reported compatibility issues.
> >
> > A list of the 33 issues resolved in this release can be found at
> > https://s.apache.org/xpxo .
> >
> > Please try out the candidate and vote +1/0/-1.
> >
> > The vote will be open for at least 72 hours. Unless objection I will try
> > to close it Monday October 8, 2018 if we have sufficient votes.
> >
> > Prior to making this announcement I made the following preflight checks:
> >
> > RAT check passes (7u80)
> > Unit test suite passes (7u80, 8u181)
> > Opened the UI in a browser, poked around
> > LTT load 1M rows with 100% verification and 20% updates (8u181)
> > ITBLL 500M rows with serverKilling monkey (8u181)
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

[jira] [Commented] (HBASE-21265) Split up TestRSGroups

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637685#comment-16637685
 ] 

Ted Yu commented on HBASE-21265:


+1

> Split up TestRSGroups
> -
>
> Key: HBASE-21265
> URL: https://issues.apache.org/jira/browse/HBASE-21265
> Project: HBase
>  Issue Type: Task
>  Components: rsgroup, test
>Affects Versions: 1.4.8
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
>  Labels: beginner, beginners
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.9
>
> Attachments: HBASE-21265-branch-1.patch
>
>
> TestRSGroups is flaky. It is stable when run in isolation but when run as 
> part of the suite with concurrent executors it can fail. The current running 
> time of this unit on my dev box is ~240 seconds (4 minutes), which is far too 
> much time. This unit should be broken up 5 to 8 ways, grouped by 
> functionality under test. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [PROPOSAL] Prepare Beam 2.8.0 release

[Ted Yu]

+1

On Wed, Oct 3, 2018 at 9:52 AM Jean-Baptiste Onofré  wrote:

> +1
>
> but we have to be fast in release process. 2.7.0 took more than 1 month
> to be cut !
>
> If no blocker, we have to just move forward.
>
> Regards
> JB
>
> On 03/10/2018 18:25, Ahmet Altay wrote:
> > Hi all,
> >
> > Release cut date for the next release is 10/10 according to Beam release
> > calendar [1]. Since the previous release is already mostly wrapped up
> > (modulo blog post), I would like to propose starting the next release on
> > time (10/10).
> >
> > Additionally I propose designating this release as the first
> > long-term-support (LTS) release [2]. This should have no impact on the
> > release process, however it would mean that we commit to patch this
> > release for the next 12 months for major issues.
> >
> > I volunteer to perform this release.
> >
> > What do you think?
> >
> > Ahmet
> >
> > [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
> > [2] https://beam.apache.org/community/policies/#releases
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

[jira] [Created] (KYLIN-3610) Upgrade Curator version

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3610:
-

 Summary: Upgrade Curator version
 Key: KYLIN-3610
 URL: https://issues.apache.org/jira/browse/KYLIN-3610
 Project: Kylin
  Issue Type: Task
Reporter: Ted Yu


Currently we use Curator 2.12.0

Latest Curator release is 4.x

We should upgrade Curator dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3610) Upgrade Curator version

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3610:
-

 Summary: Upgrade Curator version
 Key: KYLIN-3610
 URL: https://issues.apache.org/jira/browse/KYLIN-3610
 Project: Kylin
  Issue Type: Task
Reporter: Ted Yu


Currently we use Curator 2.12.0

Latest Curator release is 4.x

We should upgrade Curator dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3585) Ineffective declaration of volatile field in SparkFactDistinct

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637139#comment-16637139
 ] 

Ted Yu commented on KYLIN-3585:
---

lgtm

> Ineffective declaration of volatile field in SparkFactDistinct
> --
>
> Key: KYLIN-3585
> URL: https://issues.apache.org/jira/browse/KYLIN-3585
> Project: Kylin
>  Issue Type: Bug
>    Reporter: Ted Yu
>Priority: Minor
>
> In CuboidStatCalculator :
> {code}
> private volatile HLLCounter[] cuboidsHLL;
> {code}
> Though the array is declared volatile, the array elements are not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21221:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt, 
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp 
> should fail because the target lock is blocked by previous put
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
>   cpService.execute(() -> {
> ...
> if (!threw) {
>   // Can't call fail() earlier because the catch would eat it.
>   fail("This cp should fail because the target lock is blocked by 
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no 
> bearing on the outcome of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [apachecon 2018] Universal metrics with apache beam

[Ted Yu]

Very interesting talk, Etienne.

Looking forward to the audio recording.

Cheers

[jira] [Commented] (KYLIN-3445) Upgrade checkstyle version to 8.6

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637033#comment-16637033
 ] 

Ted Yu commented on KYLIN-3445:
---

Can this be resolved ?

> Upgrade checkstyle version to 8.6
> -
>
> Key: KYLIN-3445
> URL: https://issues.apache.org/jira/browse/KYLIN-3445
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>    Reporter: Ted Yu
>Priority: Minor
> Fix For: v2.6.0
>
>
> We should upgrade checkstyle version to 8.6+ so that we can use the "match 
> violation message to this regex" feature for suppression. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3447) Upgrade zookeeper to 3.4.13

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3447:
--
Description: 
zookeeper 3.4.13 is being released with the following fixes:

ZOOKEEPER-2959 fixes data loss when observer is used
ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / 
cloud)
environment

  was:
zookeeper 3.4.13 is being released with the following fixes:

ZOOKEEPER-2959 fixes data loss when observer is used

ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / 
cloud)
environment


> Upgrade zookeeper to 3.4.13
> ---
>
> Key: KYLIN-3447
> URL: https://issues.apache.org/jira/browse/KYLIN-3447
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Major
>
> zookeeper 3.4.13 is being released with the following fixes:
> ZOOKEEPER-2959 fixes data loss when observer is used
> ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container 
> / cloud)
> environment



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: welcome a new batch of committers

[Ted Yu]

Congratulations to all !
 Original message From: Jungtaek Lim  Date: 
10/3/18  2:41 AM  (GMT-08:00) To: Marco Gaido  Cc: dev 
 Subject: Re: welcome a new batch of committers 
Congrats all! You all deserved it.
On Wed, 3 Oct 2018 at 6:35 PM Marco Gaido  wrote:
Congrats you all!

Il giorno mer 3 ott 2018 alle ore 11:29 Liang-Chi Hsieh  ha 
scritto:


Congratulations to all new committers!





rxin wrote

> Hi all,

> 

> The Apache Spark PMC has recently voted to add several new committers to

> the project, for their contributions:

> 

> - Shane Knapp (contributor to infra)

> - Dongjoon Hyun (contributor to ORC support and other parts of Spark)

> - Kazuaki Ishizaki (contributor to Spark SQL)

> - Xingbo Jiang (contributor to Spark Core and SQL)

> - Yinan Li (contributor to Spark on Kubernetes)

> - Takeshi Yamamuro (contributor to Spark SQL)

> 

> Please join me in welcoming them!











--

Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/



-

To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[jira] [Commented] (HBASE-20690) Moving table to target rsgroup needs to handle TableStateNotFoundException

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636295#comment-16636295
 ] 

Ted Yu commented on HBASE-20690:


Triggered QA run #14566

> Moving table to target rsgroup needs to handle TableStateNotFoundException
> --
>
> Key: HBASE-20690
> URL: https://issues.apache.org/jira/browse/HBASE-20690
> Project: HBase
>  Issue Type: Bug
>    Reporter: Ted Yu
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-20690.master.001.patch, 
> HBASE-20690.master.002.patch
>
>
> This is related code:
> {code}
>   if (targetGroup != null) {
> for (TableName table: tables) {
>   if (master.getAssignmentManager().isTableDisabled(table)) {
> LOG.debug("Skipping move regions because the table" + table + " 
> is disabled.");
> continue;
>   }
> {code}
> In a stack trace [~rmani] showed me:
> {code}
> 2018-06-06 07:10:44,893 ERROR 
> [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=2] 
> master.TableStateManager: Unable to get table demo:tbl1 state
> org.apache.hadoop.hbase.master.TableStateManager$TableStateNotFoundException: 
> demo:tbl1
> at 
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:193)
> at 
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:143)
> at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.isTableDisabled(AssignmentManager.java:346)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveTables(RSGroupAdminServer.java:407)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.assignTableToGroup(RSGroupAdminEndpoint.java:447)
> at 
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.postCreateTable(RSGroupAdminEndpoint.java:470)
> at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$12.call(MasterCoprocessorHost.java:334)
> at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$12.call(MasterCoprocessorHost.java:331)
> at 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:540)
> at 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:614)
> at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postCreateTable(MasterCoprocessorHost.java:331)
> at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:1768)
> at 
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1750)
> at 
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:593)
> at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logic should take potential TableStateNotFoundException into account.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3609) NPE from QueryMetricsFacade#updateMetricsToReservoir

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3609:
-

 Summary: NPE from QueryMetricsFacade#updateMetricsToReservoir
 Key: KYLIN-3609
 URL: https://issues.apache.org/jira/browse/KYLIN-3609
 Project: Kylin
  Issue Type: Bug
Reporter: Ted Yu


When running test suite, I saw the following in test output:
{code}
2018-10-02 20:59:20,415 WARN  [Query c8f42f2e-8c77-2bfc-97ab-053fbeb7c86e-1] 
service.QueryService:423 : Write metric error.
java.lang.NullPointerException
  at 
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:148)
  at 
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:74)
  at 
org.apache.kylin.rest.service.QueryService.recordMetric(QueryService.java:505)
  at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:421)
  at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:353)
  at 
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
  at 
org.apache.kylin.rest.controller.QueryControllerTest.testQueryException(QueryControllerTest.java:63)
{code}
It seems sqlResponse.getResults() returned null.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3609) NPE from QueryMetricsFacade#updateMetricsToReservoir

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3609:
-

 Summary: NPE from QueryMetricsFacade#updateMetricsToReservoir
 Key: KYLIN-3609
 URL: https://issues.apache.org/jira/browse/KYLIN-3609
 Project: Kylin
  Issue Type: Bug
Reporter: Ted Yu


When running test suite, I saw the following in test output:
{code}
2018-10-02 20:59:20,415 WARN  [Query c8f42f2e-8c77-2bfc-97ab-053fbeb7c86e-1] 
service.QueryService:423 : Write metric error.
java.lang.NullPointerException
  at 
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:148)
  at 
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:74)
  at 
org.apache.kylin.rest.service.QueryService.recordMetric(QueryService.java:505)
  at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:421)
  at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:353)
  at 
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
  at 
org.apache.kylin.rest.controller.QueryControllerTest.testQueryException(QueryControllerTest.java:63)
{code}
It seems sqlResponse.getResults() returned null.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3608) Move dependency versions to top level pom properties

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3608:
-

 Summary: Move dependency versions to top level pom properties
 Key: KYLIN-3608
 URL: https://issues.apache.org/jira/browse/KYLIN-3608
 Project: Kylin
  Issue Type: Task
Reporter: Ted Yu


There are some non-top level pom.xml files where dependency version is 
referenced directly.

core-common/pom.xml is an example.

We should move all dependency versions to top level pom properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3608) Move dependency versions to top level pom properties

[Ted Yu (JIRA)]

Ted Yu created KYLIN-3608:
-

 Summary: Move dependency versions to top level pom properties
 Key: KYLIN-3608
 URL: https://issues.apache.org/jira/browse/KYLIN-3608
 Project: Kylin
  Issue Type: Task
Reporter: Ted Yu


There are some non-top level pom.xml files where dependency version is 
referenced directly.

core-common/pom.xml is an example.

We should move all dependency versions to top level pom properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMBARI-23288) stateWatcherClient should be closed upon return from OutputSolr#createSolrStateWatcher

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMBARI-23288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated AMBARI-23288:

Description: 
{code}
CloudSolrClient stateWatcherClient = createSolrClient();
{code}

stateWatcherClient should be closed upon return from the method.

  was:
{code}
CloudSolrClient stateWatcherClient = createSolrClient();
{code}
stateWatcherClient should be closed upon return from the method.


> stateWatcherClient should be closed upon return from 
> OutputSolr#createSolrStateWatcher
> --
>
> Key: AMBARI-23288
> URL: https://issues.apache.org/jira/browse/AMBARI-23288
> Project: Ambari
>  Issue Type: Bug
>    Reporter: Ted Yu
>Priority: Minor
>
> {code}
> CloudSolrClient stateWatcherClient = createSolrClient();
> {code}
> stateWatcherClient should be closed upon return from the method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AMBARI-24032) Ambari should be selective about which HBase components that need to be restarted

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AMBARI-24032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated AMBARI-24032:

Description: 
For hbase config changes that only involve hbase master, Ambari should 
explicitly say so after the change.

One example of such config change is for hbase.master.loadbalancer.class and 
hbase.coprocessor.master.classes

  was:
For hbase config changes that only involve hbase master, Ambari should 
explicitly say so after the change.


One example of such config change is for hbase.master.loadbalancer.class and 
hbase.coprocessor.master.classes


> Ambari should be selective about which HBase components that need to be 
> restarted
> -
>
> Key: AMBARI-24032
> URL: https://issues.apache.org/jira/browse/AMBARI-24032
> Project: Ambari
>  Issue Type: Bug
>    Reporter: Ted Yu
>Priority: Major
>
> For hbase config changes that only involve hbase master, Ambari should 
> explicitly say so after the change.
> One example of such config change is for hbase.master.loadbalancer.class and 
> hbase.coprocessor.master.classes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635659#comment-16635659
 ] 

Ted Yu commented on KYLIN-3586:
---

There are still a few instances of valueOf calls.
e.g. in Tuple.java :
{code}
return 
epicDaysToMillis(Integer.valueOf(row.getValue(partitionCol).toString()));
{code}

> Boxing/unboxing to parse a primitive is suboptimal
> --
>
> Key: KYLIN-3586
> URL: https://issues.apache.org/jira/browse/KYLIN-3586
> Project: Kylin
>  Issue Type: Bug
>    Reporter: Ted Yu
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> An example is from HBaseLookupRowEncoder:
> {code}
> int valIdx = Integer.valueOf(Bytes.toString(qualifier));
> {code}
> valueOf returns an Integer object which would be unboxed and assigned to 
> valIdx.
> Integer.parseInt() should be used instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.7

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-2517:
--
Description: 
There have been major enhancements / bug fixes since the hbase 1.1.1 release.

This issue is to upgrade to 1.4.7 release.

  was:
There have been major enhancements / bug fixes since the hbase 1.1.1 release.

This issue is to upgrade to 1.4.6 release.


> Upgrade hbase dependency to 1.4.7
> -
>
> Key: KYLIN-2517
> URL: https://issues.apache.org/jira/browse/KYLIN-2517
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Major
>
> There have been major enhancements / bug fixes since the hbase 1.1.1 release.
> This issue is to upgrade to 1.4.7 release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3046) Consider introducing log4j-extras

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3046:
--
Description: 
log4j-extras allows log rotation as well as compression.

https://logging.apache.org/log4j/extras/download.html

We should consider using log4j-extras.

  was:
log4j-extras allows log rotation as well as compression.


https://logging.apache.org/log4j/extras/download.html

We should consider using log4j-extras.


> Consider introducing log4j-extras 
> --
>
> Key: KYLIN-3046
> URL: https://issues.apache.org/jira/browse/KYLIN-3046
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Major
>  Labels: log
> Fix For: Backlog
>
>
> log4j-extras allows log rotation as well as compression.
> https://logging.apache.org/log4j/extras/download.html
> We should consider using log4j-extras.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.7

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-2517:
--
Summary: Upgrade hbase dependency to 1.4.7  (was: Upgrade hbase dependency 
to 1.4.6)

> Upgrade hbase dependency to 1.4.7
> -
>
> Key: KYLIN-2517
> URL: https://issues.apache.org/jira/browse/KYLIN-2517
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Major
>
> There have been major enhancements / bug fixes since the hbase 1.1.1 release.
> This issue is to upgrade to 1.4.6 release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-7588) Document RocksDB tuning for spinning disks

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258309#comment-16258309
 ] 

Ted Yu edited comment on FLINK-7588 at 10/2/18 1:56 AM:


bq. Be careful about whether you have enough memory to keep all bloom filters

Other than the above being tricky, the other guidelines are actionable .


was (Author: yuzhih...@gmail.com):
bq. Be careful about whether you have enough memory to keep all bloom filters

Other than the above being tricky, the other guidelines are actionable.

> Document RocksDB tuning for spinning disks
> --
>
> Key: FLINK-7588
> URL: https://issues.apache.org/jira/browse/FLINK-7588
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>    Reporter: Ted Yu
>Priority: Major
>  Labels: performance
>
> In docs/ops/state/large_state_tuning.md , it was mentioned that:
> bq. the default configuration is tailored towards SSDs and performs 
> suboptimal on spinning disks
> We should add recommendation targeting spinning disks:
> https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide#difference-of-spinning-disk



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9824) Support IPv6 literal

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-9824:
--
Description: 
Currently we use colon as separator when parsing host and port.

We should support the usage of IPv6 literals in parsing.

  was:
Currently we use colon as separator when parsing host and port.


We should support the usage of IPv6 literals in parsing.


> Support IPv6 literal
> 
>
> Key: FLINK-9824
> URL: https://issues.apache.org/jira/browse/FLINK-9824
> Project: Flink
>  Issue Type: Bug
>  Components: Network
>    Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> Currently we use colon as separator when parsing host and port.
> We should support the usage of IPv6 literals in parsing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Hadoop Ozone 0.2.1-alpha release

[Ted Yu]

Are the artifacts published on maven ?

I did a quick search but didn't find anything.

Cheers

On Mon, Oct 1, 2018 at 5:24 PM Elek, Marton  wrote:

>
> It gives me great pleasure to announce that the Apache Hadoop community
> has voted to release Apache Hadoop Ozone 0.2.1-alpha.
>
> Apache Hadoop Ozone is an object store for Hadoop built using Hadoop
> Distributed Data Store.
>
> For more information and to download, please check
>
> https://hadoop.apache.org/ozone
>
> Note: This release is alpha quality, it's not recommended to use in
> production.
>
> Many thanks to everyone who contributed to the release, and everyone in
> the Apache Hadoop community! The release is a result of work from many
> contributors. Thank you for all of them.
>
> On behalf of the Hadoop community,
> Márton Elek
>
>
> ps: Hadoop Ozone and HDDS are released separately from the main Hadoop
> releases, this release doesn't include new Hadoop Yarn/Mapreduce/Hdfs
> versions.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Comment Edited] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634744#comment-16634744
 ] 

Ted Yu edited comment on HBASE-21221 at 10/1/18 11:05 PM:
--

I noticed that the current test would pass even if MultiRowMutationEndpoint is 
not registered.
In the test output:
{code}
2018-10-01 15:55:15,749 DEBUG [hconnection-0x589a90eb-shared-pool13-t1] 
client.TestFromClientSide3(855): 
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: 
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered 
coprocessor service found for MultiRowMutationService in region 
testMultiRowMutations,,1538434514918.8d59d9ae0e4652161a3048075502367a.
  at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8223)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2484)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2466)
  at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
Attaching an addendum to exclude this scenario.


was (Author: yuzhih...@gmail.com):
I noticed that the current test would pass even if MultiRowMutationEndpoint is 
not registered.
In the test output:
{code}
2018-10-01 15:55:15,749 DEBUG [hconnection-0x589a90eb-shared-pool13-t1] 
client.TestFromClientSide3(855):  ted
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: 
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered 
coprocessor service found for MultiRowMutationService in region 
testMultiRowMutations,,1538434514918.8d59d9ae0e4652161a3048075502367a.
  at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8223)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2484)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2466)
  at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
Attaching an addendum to exclude this scenario.

> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt, 
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1"

[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21221:
---
Status: Patch Available  (was: Reopened)

> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt, 
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp 
> should fail because the target lock is blocked by previous put
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
>   cpService.execute(() -> {
> ...
> if (!threw) {
>   // Can't call fail() earlier because the catch would eat it.
>   fail("This cp should fail because the target lock is blocked by 
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no 
> bearing on the outcome of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reopened HBASE-21221:


> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt, 
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp 
> should fail because the target lock is blocked by previous put
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
>   cpService.execute(() -> {
> ...
> if (!threw) {
>   // Can't call fail() earlier because the catch would eat it.
>   fail("This cp should fail because the target lock is blocked by 
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no 
> bearing on the outcome of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21221:
---
Attachment: 21221.addendum.txt

> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt, 
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp 
> should fail because the target lock is blocked by previous put
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
>   cpService.execute(() -> {
> ...
> if (!threw) {
>   // Can't call fail() earlier because the catch would eat it.
>   fail("This cp should fail because the target lock is blocked by 
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no 
> bearing on the outcome of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reopened HBASE-21221:


> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.addendum.txt, 21221.v10.txt, 21221.v11.txt, 
> 21221.v12.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp 
> should fail because the target lock is blocked by previous put
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
>   cpService.execute(() -> {
> ...
> if (!threw) {
>   // Can't call fail() earlier because the catch would eat it.
>   fail("This cp should fail because the target lock is blocked by 
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no 
> bearing on the outcome of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634744#comment-16634744
 ] 

Ted Yu commented on HBASE-21221:


I noticed that the current test would pass even if MultiRowMutationEndpoint is 
not registered.
In the test output:
{code}
2018-10-01 15:55:15,749 DEBUG [hconnection-0x589a90eb-shared-pool13-t1] 
client.TestFromClientSide3(855):  ted
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: 
org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered 
coprocessor service found for MultiRowMutationService in region 
testMultiRowMutations,,1538434514918.8d59d9ae0e4652161a3048075502367a.
  at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8223)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2484)
  at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2466)
  at 
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
Attaching an addendum to exclude this scenario.

> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.v10.txt, 21221.v11.txt, 21221.v12.txt, 
> 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in 
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region 
> 089bdfa75f44d88e596479038a6da18b
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
>   at 
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp 
> should fail because the target lock is blocked by previous put
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
>   cpService.execute(() -> {
> ...
> if (!threw) {
>   // Can't call fail() earlier because the catch would eat it.
>   fail("This cp should fail because the target lock is blocked by 
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no 
> bearing on the outcome of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-7276) Consider using re2j to speed up regex operations

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634727#comment-16634727
 ] 

Ted Yu commented on KAFKA-7276:
---

For #1, see the following:
http://hadoop.apache.org/docs/r3.0.2/api/org/apache/hadoop/metrics2/filter/GlobFilter.html#compile-java.lang.String-

It is used by hadoop.



> Consider using re2j to speed up regex operations
> 
>
> Key: KAFKA-7276
> URL: https://issues.apache.org/jira/browse/KAFKA-7276
> Project: Kafka
>  Issue Type: Task
>  Components: packaging
>    Reporter: Ted Yu
>Assignee: kevin.chen
>Priority: Major
>
> https://github.com/google/re2j
> re2j claims to do linear time regular expression matching in Java.
> Its benefit is most obvious for deeply nested regex (such as a | b | c | d).
> We should consider using re2j to speed up regex operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6303) Potential lack of synchronization in NioEchoServer#AcceptorThread

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333251#comment-16333251
 ] 

Ted Yu edited comment on KAFKA-6303 at 10/1/18 10:42 PM:
-

+1 from me.


was (Author: yuzhih...@gmail.com):
+1 from me .

> Potential lack of synchronization in NioEchoServer#AcceptorThread
> -
>
> Key: KAFKA-6303
> URL: https://issues.apache.org/jira/browse/KAFKA-6303
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>    Reporter: Ted Yu
>Assignee: siva santhalingam
>Priority: Minor
>
> In the run() method:
> {code}
> SocketChannel socketChannel = 
> ((ServerSocketChannel) key.channel()).accept();
> socketChannel.configureBlocking(false);
> newChannels.add(socketChannel);
> {code}
> Modification to newChannels should be protected by synchronized block.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-21258:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.2.0
   1.5.0
   3.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the reviews.

> Add resetting of flags for RS Group pre/post hooks in TestRSGroups
> --
>
> Key: HBASE-21258
> URL: https://issues.apache.org/jira/browse/HBASE-21258
> Project: HBase
>  Issue Type: Test
>    Reporter: Ted Yu
>    Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt, 
> 21258.branch-2.v1.patch, 21258.v1.txt
>
>
> Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS 
> Group pre/post hooks in TestRSGroups was absent.
> This issue is to add the resetting of these flags before each subtest starts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-9924) Upgrade zookeeper to 3.4.13

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-9924:
--
Description: 
zookeeper 3.4.13 is being released.

ZOOKEEPER-2959 fixes data loss when observer is used
ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / 
cloud) environment

  was:
zookeeper 3.4.13 is being released.

ZOOKEEPER-2959 fixes data loss when observer is used

ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / 
cloud) environment


> Upgrade zookeeper to 3.4.13
> ---
>
> Key: FLINK-9924
> URL: https://issues.apache.org/jira/browse/FLINK-9924
> Project: Flink
>  Issue Type: Task
>    Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Major
>
> zookeeper 3.4.13 is being released.
> ZOOKEEPER-2959 fixes data loss when observer is used
> ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container 
> / cloud) environment



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (FLINK-10391) MillisOfDay is used in place of instant for LocalTime ctor in AvroKryoSerializerUtils

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated FLINK-10391:
---
Description: 
>From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value 
>from LocalTime.
For read method:

{code}
  final int time = input.readInt(true);
  return new LocalTime(time, 
ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
{code}
It seems 
http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
 should be used instead.

  was:
>From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value 
>from LocalTime.
For read method:
{code}
  final int time = input.readInt(true);
  return new LocalTime(time, 
ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
{code}
It seems 
http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
 should be used instead.


> MillisOfDay is used in place of instant for LocalTime ctor in 
> AvroKryoSerializerUtils
> -
>
> Key: FLINK-10391
> URL: https://issues.apache.org/jira/browse/FLINK-10391
> Project: Flink
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value 
> from LocalTime.
> For read method:
> {code}
>   final int time = input.readInt(true);
>   return new LocalTime(time, 
> ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
> {code}
> It seems 
> http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
>  should be used instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3556) Interned string should not be used as lock object

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3556:
--
Description: 
In JDBCResourceDAO :

{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with 
other part of code.

  was:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with 
other part of code.


> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee:  Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with 
> other part of code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3290) Avoid calling Class#newInstance

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3290:
--
Description: 
Class#newInstance is deprecated starting in Java 9 - 
https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
undeclared checked exceptions.


The suggested replacement is getDeclaredConstructor().newInstance(), which 
wraps the checked exceptions in InvocationException.

  was:
Class#newInstance is deprecated starting in Java 9 - 
https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
undeclared checked exceptions.

The suggested replacement is getDeclaredConstructor().newInstance(), which 
wraps the checked exceptions in InvocationException.


> Avoid calling Class#newInstance
> ---
>
> Key: KYLIN-3290
> URL: https://issues.apache.org/jira/browse/KYLIN-3290
> Project: Kylin
>  Issue Type: Task
>    Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Minor
>  Labels: jdk
> Fix For: v2.6.0
>
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624443#comment-16624443
 ] 

Ted Yu edited comment on KYLIN-3417 at 10/1/18 4:24 PM:


For phase I, we don't need to use Optimistic read lock.


was (Author: yuzhih...@gmail.com):
For phase I, we don't need to use Optimistic read lock

> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold 
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.17.0

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-2650:
--
Summary: Update to Apache Calcite Avatica 1.17.0  (was: Update to Apache 
Calcite Avatica 1.12.0)

> Update to Apache Calcite Avatica 1.17.0
> ---
>
> Key: KYLIN-2650
> URL: https://issues.apache.org/jira/browse/KYLIN-2650
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Minor
>
> Apache Calcite Avatica 1.17.0 was released mid-July
> https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released
> This issue upgrades Avatica dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.12.0

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-2650:
--
Description: 
Apache Calcite Avatica 1.17.0 was released mid-July

https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released

This issue upgrades Avatica dependency.

  was:
Apache Calcite Avatica 1.12.0 has just been released.

https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released

This issue upgrades Avatica dependency.


> Update to Apache Calcite Avatica 1.12.0
> ---
>
> Key: KYLIN-2650
> URL: https://issues.apache.org/jira/browse/KYLIN-2650
> Project: Kylin
>  Issue Type: Improvement
>    Reporter: Ted Yu
>Priority: Minor
>
> Apache Calcite Avatica 1.17.0 was released mid-July
> https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released
> This issue upgrades Avatica dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] SPARK 2.4.0 (RC2)

[Ted Yu]

+1
 Original message From: Denny Lee  Date: 
9/30/18  10:30 PM  (GMT-08:00) To: Stavros Kontopoulos 
 Cc: Sean Owen , Wenchen 
Fan , dev  Subject: Re: [VOTE] SPARK 
2.4.0 (RC2) 
+1 (non-binding)


On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos 
 wrote:
+1
Stavros
On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen  wrote:
+1, with comments:



There are 5 critical issues for 2.4, and no blockers:

SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4

SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs

SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella

SPARK-25326 ML, Graph 2.4 QA: Programming guide update and migration guide

SPARK-25323 ML 2.4 QA: API: Python API coverage



Xiangrui, is SPARK-25378 important enough we need to get it into 2.4?



I found two issues resolved for 2.4.1 that got into this RC, so marked

them as resolved in 2.4.0.



I checked the licenses and notice and they look correct now in source

and binary builds.



The 2.12 artifacts are as I'd expect.



I ran all tests for 2.11 and 2.12 and they pass with -Pyarn

-Pkubernetes -Pmesos -Phive -Phadoop-2.7 -Pscala-2.12.









On Thu, Sep 27, 2018 at 10:00 PM Wenchen Fan  wrote:

>

> Please vote on releasing the following candidate as Apache Spark version 
> 2.4.0.

>

> The vote is open until October 1 PST and passes if a majority +1 PMC votes 
> are cast, with

> a minimum of 3 +1 votes.

>

> [ ] +1 Release this package as Apache Spark 2.4.0

> [ ] -1 Do not release this package because ...

>

> To learn more about Apache Spark, please see http://spark.apache.org/

>

> The tag to be voted on is v2.4.0-rc2 (commit 
> 42f25f309e91c8cde1814e3720099ac1e64783da):

> https://github.com/apache/spark/tree/v2.4.0-rc2

>

> The release files, including signatures, digests, etc. can be found at:

> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-bin/

>

> Signatures used for Spark RCs can be found in this file:

> https://dist.apache.org/repos/dist/dev/spark/KEYS

>

> The staging repository for this release can be found at:

> https://repository.apache.org/content/repositories/orgapachespark-1287

>

> The documentation corresponding to this release can be found at:

> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc2-docs/

>

> The list of bug fixes going into 2.4.0 can be found at the following URL:

> https://issues.apache.org/jira/projects/SPARK/versions/2.4.0

>

> FAQ

>

> =

> How can I help test this release?

> =

>

> If you are a Spark user, you can help us test this release by taking

> an existing Spark workload and running on this release candidate, then

> reporting any regressions.

>

> If you're working in PySpark you can set up a virtual env and install

> the current RC and see if anything important breaks, in the Java/Scala

> you can add the staging repository to your projects resolvers and test

> with the RC (make sure to clean up the artifact cache before/after so

> you don't end up building with a out of date RC going forward).

>

> ===

> What should happen to JIRA tickets still targeting 2.4.0?

> ===

>

> The current list of open tickets targeted at 2.4.0 can be found at:

> https://issues.apache.org/jira/projects/SPARK and search for "Target 
> Version/s" = 2.4.0

>

> Committers should look at those and triage. Extremely important bug

> fixes, documentation, and API tweaks that impact compatibility should

> be worked on immediately. Everything else please retarget to an

> appropriate release.

>

> ==

> But my bug isn't fixed?

> ==

>

> In order to make timely releases, we will typically not hold the

> release unless the bug in question is a regression from the previous

> release. That being said, if there is something which is a regression

> that has not been correctly targeted please ping me or a committer to

> help target the issue.



-

To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[jira] [Comment Edited] (KYLIN-3310) Use lint for maven-compiler-plugin

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560940#comment-16560940
 ] 

Ted Yu edited comment on KYLIN-3310 at 10/1/18 1:40 AM:


Thanks, Jiatao.


was (Author: yuzhih...@gmail.com):
Thanks, Jiatao .

> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>    Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.6.0
>
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
>   maven-compiler-plugin
>   ${maven-compiler-plugin.version}
>   
> 1.8
> 1.8
> 
>   -Xlint:all
>   ${compiler.error.flag}
>   
>   -Xlint:-options
>   
>   -Xlint:-cast
>   -Xlint:-deprecation
>   -Xlint:-processing
>   -Xlint:-rawtypes
>   -Xlint:-serial
>   -Xlint:-try
>   -Xlint:-unchecked
>   -Xlint:-varargs
>   
>   
>   
> 
> true
> 
> false
>   
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)