[jira] [Updated] (FLINK-9363) Bump up the Jackson version
[ https://issues.apache.org/jira/browse/FLINK-9363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-9363: -- Description: CVE's for Jackson : CVE-2017-17485 CVE-2018-5968 CVE-2018-7489 We can upgrade to 2.9.5 was: CVE's for Jackson : CVE-2017-17485 CVE-2018-5968 CVE-2018-7489 We can upgrade to 2.9.5 > Bump up the Jackson version > --- > > Key: FLINK-9363 > URL: https://issues.apache.org/jira/browse/FLINK-9363 > Project: Flink > Issue Type: Improvement > Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > Labels: security > > CVE's for Jackson : > CVE-2017-17485 > CVE-2018-5968 > CVE-2018-7489 > We can upgrade to 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21230) BackupUtils#checkTargetDir doesn't compose error message correctly
Ted Yu created HBASE-21230:
--
Summary: BackupUtils#checkTargetDir doesn't compose error message
correctly
Key: HBASE-21230
URL: https://issues.apache.org/jira/browse/HBASE-21230
Project: HBase
Issue Type: Bug
Components: backuprestore
Reporter: Ted Yu
Here is related code:
{code}
String expMsg = e.getMessage();
String newMsg = null;
if (expMsg.contains("No FileSystem for scheme")) {
newMsg =
"Unsupported filesystem scheme found in the backup target url.
Error Message: "
+ newMsg;
{code}
I think the intention was to concatenate expMsg at the end of newMsg.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21230) BackupUtils#checkTargetDir doesn't compose error message correctly
Ted Yu created HBASE-21230:
--
Summary: BackupUtils#checkTargetDir doesn't compose error message
correctly
Key: HBASE-21230
URL: https://issues.apache.org/jira/browse/HBASE-21230
Project: HBase
Issue Type: Bug
Components: backuprestore
Reporter: Ted Yu
Here is related code:
{code}
String expMsg = e.getMessage();
String newMsg = null;
if (expMsg.contains("No FileSystem for scheme")) {
newMsg =
"Unsupported filesystem scheme found in the backup target url.
Error Message: "
+ newMsg;
{code}
I think the intention was to concatenate expMsg at the end of newMsg.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Thanks for the review, Mingliang.
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: 21221.v10.txt, 21221.v11.txt, 21221.v12.txt,
> 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21228) Memory leak since AbstractFSWAL caches Thread object and never clean later
[
https://issues.apache.org/jira/browse/HBASE-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627554#comment-16627554
]
Ted Yu commented on HBASE-21228:
+1, pending QA
> Memory leak since AbstractFSWAL caches Thread object and never clean later
> --
>
> Key: HBASE-21228
> URL: https://issues.apache.org/jira/browse/HBASE-21228
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.2, 1.4.7
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: HBASE-21228.branch-2.0.001.patch
>
>
> In AbstractFSWAL(FSHLog in branch-1), we have a map caches thread and
> SyncFutures.
> {code}
> /**
>* Map of {@link SyncFuture}s keyed by Handler objects. Used so we reuse
> SyncFutures.
>*
>* TODO: Reuse FSWALEntry's rather than create them anew each time as we do
> SyncFutures here.
>*
>* TODO: Add a FSWalEntry and SyncFuture as thread locals on handlers
> rather than have them get
>* them from this Map?
>*/
> private final ConcurrentMap syncFuturesByHandler;
> {code}
> A colleague of mine find a memory leak case caused by this map.
> Every thread who writes WAL will be cached in this map, And no one will clean
> the threads in the map even after the thread is dead.
> In one of our customer's cluster, we noticed that even though there is no
> requests, the heap of the RS is almost full and CMS GC was triggered every
> second.
> We dumped the heap and then found out there were more than 30 thousands
> threads with Terminated state. which are all cached in this map above.
> Everything referenced in these threads were leaked. Most of the threads are:
> 1.PostOpenDeployTasksThread, which will write Open Region mark in WAL
> 2. hconnection-0x1f838e31-shared--pool, which are used to write index short
> circuit(Phoenix), and WAL will be write and sync in these threads.
> 3. Index writer thread(Phoenix), which referenced by
> RegionCoprocessorHost$RegionEnvironment then by HRegion and finally been
> referenced by PostOpenDeployTasksThread.
> We should turn this map into a thread local one, let JVM GC the terminated
> thread for us.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-5863) Potential null dereference in DistributedHerder#reconfigureConnector()
[
https://issues.apache.org/jira/browse/KAFKA-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KAFKA-5863:
--
Description:
Here is the call chain:
{code}
RestServer.httpRequest(reconfigUrl, "POST",
taskProps, null);
{code}
In httpRequest():
{code}
} else if (responseCode >= 200 && responseCode < 300) {
InputStream is = connection.getInputStream();
T result = JSON_SERDE.readValue(is, responseFormat);
{code}
For readValue():
{code}
public T readValue(InputStream src, TypeReference valueTypeRef)
throws IOException, JsonParseException, JsonMappingException
{
return (T) _readMapAndClose(_jsonFactory.createParser(src),
_typeFactory.constructType(valueTypeRef));
{code}
Then there would be NPE in constructType():
{code}
public JavaType constructType(TypeReference typeRef)
{
// 19-Oct-2015, tatu: Simpler variant like so should work
return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS);
{code}
was:
Here is the call chain:
{code}
RestServer.httpRequest(reconfigUrl, "POST",
taskProps, null);
{code}
In httpRequest():
{code}
} else if (responseCode >= 200 && responseCode < 300) {
InputStream is = connection.getInputStream();
T result = JSON_SERDE.readValue(is, responseFormat);
{code}
For readValue():
{code}
public T readValue(InputStream src, TypeReference valueTypeRef)
throws IOException, JsonParseException, JsonMappingException
{
return (T) _readMapAndClose(_jsonFactory.createParser(src),
_typeFactory.constructType(valueTypeRef));
{code}
Then there would be NPE in constructType():
{code}
public JavaType constructType(TypeReference typeRef)
{
// 19-Oct-2015, tatu: Simpler variant like so should work
return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS);
{code}
> Potential null dereference in DistributedHerder#reconfigureConnector()
> --
>
> Key: KAFKA-5863
> URL: https://issues.apache.org/jira/browse/KAFKA-5863
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Ted Yu
>Priority: Minor
>
> Here is the call chain:
> {code}
> RestServer.httpRequest(reconfigUrl, "POST",
> taskProps, null);
> {code}
> In httpRequest():
> {code}
> } else if (responseCode >= 200 && responseCode < 300) {
> InputStream is = connection.getInputStream();
> T result = JSON_SERDE.readValue(is, responseFormat);
> {code}
> For readValue():
> {code}
> public T readValue(InputStream src, TypeReference valueTypeRef)
> throws IOException, JsonParseException, JsonMappingException
> {
> return (T) _readMapAndClose(_jsonFactory.createParser(src),
> _typeFactory.constructType(valueTypeRef));
> {code}
> Then there would be NPE in constructType():
> {code}
> public JavaType constructType(TypeReference typeRef)
> {
> // 19-Oct-2015, tatu: Simpler variant like so should work
> return _fromAny(null, typeRef.getType(), EMPTY_BINDINGS);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Attachment: 21221.v12.txt
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Attachments: 21221.v10.txt, 21221.v11.txt, 21221.v12.txt,
> 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-14163) zookeeper session timeout for hbase should take zookeeper tickTime into account
[ https://issues.apache.org/jira/browse/AMBARI-14163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated AMBARI-14163: Description: With tickTime=2000 in zoo.cfg, I tried to set zookeeper.session.timeout value of 1 min 40 seconds. The change was accepted. However, such timeout is not reachable (it is > 20 times tickTime). Ambari should detect such scenario and warn user. was: With tickTime=2000 in zoo.cfg, I tried to set zookeeper.session.timeout value of 1 min 40 seconds. The change was accepted. However, such timeout is not reachable (it is > 20 times tickTime). Ambari should detect such scenario and warn user. > zookeeper session timeout for hbase should take zookeeper tickTime into > account > --- > > Key: AMBARI-14163 > URL: https://issues.apache.org/jira/browse/AMBARI-14163 > Project: Ambari > Issue Type: Bug >Reporter: Ted Yu >Priority: Major > > With tickTime=2000 in zoo.cfg, I tried to set zookeeper.session.timeout value > of 1 min 40 seconds. > The change was accepted. > However, such timeout is not reachable (it is > 20 times tickTime). > Ambari should detect such scenario and warn user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-17346) Dependent components should be shutdown before stopping hdfs
[ https://issues.apache.org/jira/browse/AMBARI-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated AMBARI-17346: Description: Sometimes admin shuts down hdfs first, then hbase. By the time hbase is shutdown, no data can be persisted (including metadata). This results in large number of inconsistencies when hbase cluster is brought back up. Before hdfs is shutdown, the components dependent on hdfs should be shutdown first. was: Sometimes admin shuts down hdfs first, then hbase. By the time hbase is shutdown, no data can be persisted (including metadata). This results in large number of inconsistencies when hbase cluster is brought back up. Before hdfs is shutdown, the components dependent on hdfs should be shutdown first. > Dependent components should be shutdown before stopping hdfs > > > Key: AMBARI-17346 > URL: https://issues.apache.org/jira/browse/AMBARI-17346 > Project: Ambari > Issue Type: Bug > Reporter: Ted Yu >Priority: Major > > Sometimes admin shuts down hdfs first, then hbase. > By the time hbase is shutdown, no data can be persisted (including metadata). > This results in large number of inconsistencies when hbase cluster is brought > back up. > Before hdfs is shutdown, the components dependent on hdfs should be shutdown > first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21219) Hbase incremental backup fails with null pointer exception
[
https://issues.apache.org/jira/browse/HBASE-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626634#comment-16626634
]
Ted Yu commented on HBASE-21219:
{code}
307 LOG.warn("Known hosts (from newestTimestamps):");
308 for (String s: newestTimestamps.keySet()) {
309 LOG.warn(s);
{code}
The above seems to be for debug purpose. Use LOG.debug ?
> Hbase incremental backup fails with null pointer exception
> --
>
> Key: HBASE-21219
> URL: https://issues.apache.org/jira/browse/HBASE-21219
> Project: HBase
> Issue Type: Bug
> Components: backuprestore
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21219-v1.patch
>
>
> hbase backup create incremental hdfs:///bkpHbase_Test/bkpHbase_Test2 -t
> bkpHbase_Test2
> 2018-09-21 15:35:31,421 INFO [main] impl.TableBackupClient: Backup
> backup_1537524313995 started at 1537524331419. 2018-09-21 15:35:31,454 INFO
> [main] impl.IncrementalBackupManager: Execute roll log procedure for
> incremental backup ... 2018-09-21 15:35:32,985 ERROR [main]
> impl.TableBackupClient: Unexpected Exception : java.lang.NullPointerException
> java.lang.NullPointerException at
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
> org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179)
> 2018-09-21 15:35:32,989 ERROR [main] impl.TableBackupClient:
> BackupId=backup_1537524313995,startts=1537524331419,failedts=1537524332989,failedphase=PREPARE_INCREMENTAL,failedmessage=null
> 2018-09-21 15:35:57,167 ERROR [main] impl.TableBackupClient: Backup
> backup_1537524313995 failed.
> Backup session finished. Status: FAILURE 2018-09-21 15:35:57,175 ERROR [main]
> backup.BackupDriver: Error running
> command-line tool java.io.IOException: java.lang.NullPointerException at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:281)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601)
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
> org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179)
> Caused by: java.lang.NullPointerException at
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276)
> ... 7 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Attachment: 21221.v11.txt
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Attachments: 21221.v10.txt, 21221.v11.txt, 21221.v7.txt,
> 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626563#comment-16626563
]
Ted Yu commented on HBASE-21221:
bq. the ex in RecordingEndpoint should be safely published
The custom endpoint is removed in patch v10. So the above is not needed.
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Attachments: 21221.v10.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Attachment: 21221.v10.txt
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Attachments: 21221.v10.txt, 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3590) Missing argument to exception message String in ModelController
[
https://issues.apache.org/jira/browse/KYLIN-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626212#comment-16626212
]
Ted Yu commented on KYLIN-3590:
---
There is similar issue in ProjectController#saveProject
> Missing argument to exception message String in ModelController
> ---
>
> Key: KYLIN-3590
> URL: https://issues.apache.org/jira/browse/KYLIN-3590
> Project: Kylin
> Issue Type: Bug
> Reporter: Ted Yu
>Priority: Major
>
> Here is related code:
> {code}
> String.format(Locale.ROOT,
> "Invalid model name %s, only letters, numbers and
> underscore " + "supported."),
> {code}
> There is no argument for '%s'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3590) Missing argument to exception message String in ModelController
Ted Yu created KYLIN-3590:
-
Summary: Missing argument to exception message String in
ModelController
Key: KYLIN-3590
URL: https://issues.apache.org/jira/browse/KYLIN-3590
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
String.format(Locale.ROOT,
"Invalid model name %s, only letters, numbers and
underscore " + "supported."),
{code}
There is no argument for '%s'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3590) Missing argument to exception message String in ModelController
Ted Yu created KYLIN-3590:
-
Summary: Missing argument to exception message String in
ModelController
Key: KYLIN-3590
URL: https://issues.apache.org/jira/browse/KYLIN-3590
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
String.format(Locale.ROOT,
"Invalid model name %s, only letters, numbers and
underscore " + "supported."),
{code}
There is no argument for '%s'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3589) Different type is used to check presence in Collection in AclPermissionUtil
Ted Yu created KYLIN-3589:
-
Summary: Different type is used to check presence in Collection in
AclPermissionUtil
Key: KYLIN-3589
URL: https://issues.apache.org/jira/browse/KYLIN-3589
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public static List transformAuthorities(Collection authorities) {
List ret = new ArrayList();
for (GrantedAuthority auth : authorities) {
if (!authorities.contains(auth.getAuthority())) {
{code}
auth.getAuthority() shouldn't be used to check presence in the Collection since
it is not of GrantedAuthority type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3589) Different type is used to check presence in Collection in AclPermissionUtil
Ted Yu created KYLIN-3589:
-
Summary: Different type is used to check presence in Collection in
AclPermissionUtil
Key: KYLIN-3589
URL: https://issues.apache.org/jira/browse/KYLIN-3589
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public static List transformAuthorities(Collection authorities) {
List ret = new ArrayList();
for (GrantedAuthority auth : authorities) {
if (!authorities.contains(auth.getAuthority())) {
{code}
auth.getAuthority() shouldn't be used to check presence in the Collection since
it is not of GrantedAuthority type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3588) Potentially duplicate put to RemoveBlackoutRealizationsRule#filters map
Ted Yu created KYLIN-3588:
-
Summary: Potentially duplicate put to
RemoveBlackoutRealizationsRule#filters map
Key: KYLIN-3588
URL: https://issues.apache.org/jira/browse/KYLIN-3588
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
IRealizationFilter filter = filters.get(conf);
if (filter == null) {
try {
Class clz =
ClassUtil.forName(conf.getQueryRealizationFilter(), IRealizationFilter.class);
filter =
clz.getConstructor(KylinConfig.class).newInstance(conf);
} catch (Exception e) {
throw new RuntimeException(e);
}
filters.put(conf, filter);
{code}
Two threads may find out there is no filter for conf, they would proceed with
creation of IRealizationFilter instances and try to put into the map.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3588) Potentially duplicate put to RemoveBlackoutRealizationsRule#filters map
Ted Yu created KYLIN-3588:
-
Summary: Potentially duplicate put to
RemoveBlackoutRealizationsRule#filters map
Key: KYLIN-3588
URL: https://issues.apache.org/jira/browse/KYLIN-3588
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
IRealizationFilter filter = filters.get(conf);
if (filter == null) {
try {
Class clz =
ClassUtil.forName(conf.getQueryRealizationFilter(), IRealizationFilter.class);
filter =
clz.getConstructor(KylinConfig.class).newInstance(conf);
} catch (Exception e) {
throw new RuntimeException(e);
}
filters.put(conf, filter);
{code}
Two threads may find out there is no filter for conf, they would proceed with
creation of IRealizationFilter instances and try to put into the map.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3587) Null value is used in IllegalStateException message
Ted Yu created KYLIN-3587:
-
Summary: Null value is used in IllegalStateException message
Key: KYLIN-3587
URL: https://issues.apache.org/jira/browse/KYLIN-3587
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In ProjectManager :
{code}
TableDesc table = metaMgr.getTableDesc(tableId, projectName);
if (table == null) {
throw new IllegalStateException("Cannot find table '" +
table + "' in metadata manager");
{code}
Since table is null, the exception message is not helpful.
It seems including tableId in the message is better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3587) Null value is used in IllegalStateException message
Ted Yu created KYLIN-3587:
-
Summary: Null value is used in IllegalStateException message
Key: KYLIN-3587
URL: https://issues.apache.org/jira/browse/KYLIN-3587
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In ProjectManager :
{code}
TableDesc table = metaMgr.getTableDesc(tableId, projectName);
if (table == null) {
throw new IllegalStateException("Cannot find table '" +
table + "' in metadata manager");
{code}
Since table is null, the exception message is not helpful.
It seems including tableId in the message is better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AMBARI-13109) Verify that max logs parameter wouldn't cause premature region flush
[ https://issues.apache.org/jira/browse/AMBARI-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120333#comment-16120333 ] Ted Yu edited comment on AMBARI-13109 at 9/24/18 3:27 PM: -- The max logs parameter plays an important role in preventing fragmentation in memstore . was (Author: yuzhih...@gmail.com): The max logs parameter plays an important role in preventing fragmentation in memstore. > Verify that max logs parameter wouldn't cause premature region flush > > > Key: AMBARI-13109 > URL: https://issues.apache.org/jira/browse/AMBARI-13109 > Project: Ambari > Issue Type: Improvement > Reporter: Ted Yu >Priority: Major > > For hbase, the following relationship should be maintained: > dfs.blocksize * 0.95 * hbase.regionserver.maxlogs (default 32) > > hbase.regionserver.global.memstore.upperLimit * HBASE_HEAPSIZE > The rationale is to avoid premature region flush due to log roll which is > triggered by insufficiently high max logs. > When user changes the values for max logs, memstore upper limit or max heap > size, we should verify that the above relationship is satisfied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-16627.
Resolution: Later
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-16627) AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table exists
[
https://issues.apache.org/jira/browse/HBASE-16627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-16627.
Resolution: Later
> AssignmentManager#isDisabledorDisablingRegionInRIT should check whether table
> exists
>
>
> Key: HBASE-16627
> URL: https://issues.apache.org/jira/browse/HBASE-16627
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
>Assignee: Stephen Yuan Jiang
>Priority: Minor
>
> [~stack] first reported this issue when he played with backup feature.
> The following exception can be observed in backup unit tests:
> {code}
> 2016-09-13 16:21:57,661 ERROR [ProcedureExecutor-3]
> master.TableStateManager(134): Unable to get table hbase:backup state
> org.apache.hadoop.hbase.TableNotFoundException: hbase:backup
> at
> org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
> at
> org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546)
> at
> org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127)
> at
> org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808)
> {code}
> AssignmentManager#isDisabledorDisablingRegionInRIT should take table
> existence into account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18580) Allow gdb to attach to selected process in docker
[ https://issues.apache.org/jira/browse/HBASE-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18580: --- Labels: debugger (was: ) > Allow gdb to attach to selected process in docker > - > > Key: HBASE-18580 > URL: https://issues.apache.org/jira/browse/HBASE-18580 > Project: HBase > Issue Type: Sub-task > Reporter: Ted Yu >Priority: Major > Labels: debugger > > In the current docker image, if you want to attach gdb to a process, you > would see: > bq. ptrace: Operation not permitted > We should provide better support for gdb in docker. > This article is a start: > https://thirld.com/blog/2016/08/15/war-stories-debugging-julia-gdb-docker/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3561) Upgrade to JUnit 5
[ https://issues.apache.org/jira/browse/KYLIN-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3561: -- Description: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 was: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 > Upgrade to JUnit 5 > -- > > Key: KYLIN-3561 > URL: https://issues.apache.org/jira/browse/KYLIN-3561 > Project: Kylin > Issue Type: Task > Reporter: Ted Yu >Priority: Major > > JUnit 5 brings multiple useful features so tests are easier to read and write. > We can bump up the dependency version and create new tests with JUnit 5 > features. > Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests > https://twitter.com/nipafx/status/1027095088059559936 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18547) Provide better cleanup of WAL related entries in backup table
[ https://issues.apache.org/jira/browse/HBASE-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18547: --- Labels: backup (was: ) > Provide better cleanup of WAL related entries in backup table > - > > Key: HBASE-18547 > URL: https://issues.apache.org/jira/browse/HBASE-18547 > Project: HBase > Issue Type: Bug > Reporter: Ted Yu >Priority: Major > Labels: backup > > I was discussing the design around incremental backup with Vladimir. > Currently if WAL file is recorded in backup table, BackupLogCleaner would > signal that the file can be cleaned up. > However, cleaner chore doesn't notify BackupLogCleaner whether the file is > actually deleted. > This means that potentially large number of WAL file entries would stay in > backup table. > This issue is to investigate better cleanup strategy for the WAL entries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18610) Provide capability to activate chaos monkey
[ https://issues.apache.org/jira/browse/HBASE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18610: --- Attachment: (was: 18610.v1.txt) > Provide capability to activate chaos monkey > --- > > Key: HBASE-18610 > URL: https://issues.apache.org/jira/browse/HBASE-18610 > Project: HBase > Issue Type: Sub-task > Reporter: Ted Yu >Priority: Major > Labels: fault-tolerance > Attachments: 18610.v4.txt, 18610.v5.txt, 18610.v6.txt, 18610.v7.txt, > 18610.v8.txt, 18610.v9.txt > > > Currently load-client runs against a cluster where region servers are stable. > We need to introduce chaos monkey so that wider coverage for read path is > exercised. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (GORA-545) Boxing/unboxing to parse a primitive is suboptimal
Ted Yu created GORA-545:
---
Summary: Boxing/unboxing to parse a primitive is suboptimal
Key: GORA-545
URL: https://issues.apache.org/jira/browse/GORA-545
Project: Apache Gora
Issue Type: Bug
Reporter: Ted Yu
An example comes from InfinispanStore.java :
{code}
splitSize = Integer.valueOf(
properties.getProperty( BUFFER_LIMIT_READ_NAME,
{code}
splitSize, an int, should be assigned by calling Integer.parseInt()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal
Ted Yu created KYLIN-3586:
-
Summary: Boxing/unboxing to parse a primitive is suboptimal
Key: KYLIN-3586
URL: https://issues.apache.org/jira/browse/KYLIN-3586
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
An example is from HBaseLookupRowEncoder:
{code}
int valIdx = Integer.valueOf(Bytes.toString(qualifier));
{code}
valueOf returns an Integer object which would be unboxed and assigned to valIdx.
Integer.parseInt() should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal
Ted Yu created KYLIN-3586:
-
Summary: Boxing/unboxing to parse a primitive is suboptimal
Key: KYLIN-3586
URL: https://issues.apache.org/jira/browse/KYLIN-3586
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
An example is from HBaseLookupRowEncoder:
{code}
int valIdx = Integer.valueOf(Bytes.toString(qualifier));
{code}
valueOf returns an Integer object which would be unboxed and assigned to valIdx.
Integer.parseInt() should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3585) Ineffective declaration of volatile field in SparkFactDistinct
Ted Yu created KYLIN-3585:
-
Summary: Ineffective declaration of volatile field in
SparkFactDistinct
Key: KYLIN-3585
URL: https://issues.apache.org/jira/browse/KYLIN-3585
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In CuboidStatCalculator :
{code}
private volatile HLLCounter[] cuboidsHLL;
{code}
Though the array is declared volatile, the array elements are not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3585) Ineffective declaration of volatile field in SparkFactDistinct
Ted Yu created KYLIN-3585:
-
Summary: Ineffective declaration of volatile field in
SparkFactDistinct
Key: KYLIN-3585
URL: https://issues.apache.org/jira/browse/KYLIN-3585
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In CuboidStatCalculator :
{code}
private volatile HLLCounter[] cuboidsHLL;
{code}
Though the array is declared volatile, the array elements are not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3584) Incorrect lazy initialization of DefaultScheduler.INSTANCE
Ted Yu created KYLIN-3584:
-
Summary: Incorrect lazy initialization of DefaultScheduler.INSTANCE
Key: KYLIN-3584
URL: https://issues.apache.org/jira/browse/KYLIN-3584
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public static DefaultScheduler getInstance() {
if (INSTANCE == null) {
INSTANCE = createInstance();
{code}
The check against may not be effective.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3584) Incorrect lazy initialization of DefaultScheduler.INSTANCE
Ted Yu created KYLIN-3584:
-
Summary: Incorrect lazy initialization of DefaultScheduler.INSTANCE
Key: KYLIN-3584
URL: https://issues.apache.org/jira/browse/KYLIN-3584
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public static DefaultScheduler getInstance() {
if (INSTANCE == null) {
INSTANCE = createInstance();
{code}
The check against may not be effective.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3583) Integer multiplication is performed where Long result is expected
Ted Yu created KYLIN-3583:
-
Summary: Integer multiplication is performed where Long result is
expected
Key: KYLIN-3583
URL: https://issues.apache.org/jira/browse/KYLIN-3583
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In RecordEventTimeDetail :
{code}
long timeStampForWeekBegin = timeStamp;
timeStampForWeekBegin -= 360 * 24 *
(calendar.get(Calendar.DAY_OF_WEEK) - 1);
{code}
Casting to long should be done first before multiplication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3583) Integer multiplication is performed where Long result is expected
Ted Yu created KYLIN-3583:
-
Summary: Integer multiplication is performed where Long result is
expected
Key: KYLIN-3583
URL: https://issues.apache.org/jira/browse/KYLIN-3583
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In RecordEventTimeDetail :
{code}
long timeStampForWeekBegin = timeStamp;
timeStampForWeekBegin -= 360 * 24 *
(calendar.get(Calendar.DAY_OF_WEEK) - 1);
{code}
Casting to long should be done first before multiplication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3582) String comparison should not use == in RecordEvent
Ted Yu created KYLIN-3582:
-
Summary: String comparison should not use == in RecordEvent
Key: KYLIN-3582
URL: https://issues.apache.org/jira/browse/KYLIN-3582
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
for (RecordReserveKeyEnum reserveKey :
RecordReserveKeyEnum.values()) {
if (reserveKey.reserveKey == key) {
{code}
equals() should be used to compare Strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3582) String comparison should not use == in RecordEvent
Ted Yu created KYLIN-3582:
-
Summary: String comparison should not use == in RecordEvent
Key: KYLIN-3582
URL: https://issues.apache.org/jira/browse/KYLIN-3582
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
for (RecordReserveKeyEnum reserveKey :
RecordReserveKeyEnum.values()) {
if (reserveKey.reserveKey == key) {
{code}
equals() should be used to compare Strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3581) compareTo method should be paired with equals method
Ted Yu created KYLIN-3581: - Summary: compareTo method should be paired with equals method Key: KYLIN-3581 URL: https://issues.apache.org/jira/browse/KYLIN-3581 Project: Kylin Issue Type: Bug Reporter: Ted Yu DoggedCubeBuilder$MergeSlot and InMemCubeBuilder$CuboidTask define compareTo method. They should define equals method as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3581) compareTo method should be paired with equals method
Ted Yu created KYLIN-3581: - Summary: compareTo method should be paired with equals method Key: KYLIN-3581 URL: https://issues.apache.org/jira/browse/KYLIN-3581 Project: Kylin Issue Type: Bug Reporter: Ted Yu DoggedCubeBuilder$MergeSlot and InMemCubeBuilder$CuboidTask define compareTo method. They should define equals method as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3580) CuboidStatsUtil#complementRowCountForMandatoryCuboids uses entrySet to add elements
Ted Yu created KYLIN-3580:
-
Summary: CuboidStatsUtil#complementRowCountForMandatoryCuboids
uses entrySet to add elements
Key: KYLIN-3580
URL: https://issues.apache.org/jira/browse/KYLIN-3580
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
sortedStatsSet.addAll(statistics.entrySet());
{code}
The entrySet() method is allowed to return a view of the underlying Map in
which a single Entry object is reused and returned during the iteration.
When iterating through such a Map, the Entry value is only valid until you
advance to the next iteration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3579) entrySet iterator should be used in BPUSCalculator
Ted Yu created KYLIN-3579:
-
Summary: entrySet iterator should be used in BPUSCalculator
Key: KYLIN-3579
URL: https://issues.apache.org/jira/browse/KYLIN-3579
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
BPUSCalculator.calculateBenefitTotal and BPUSCalculator.initCuboidAggCostMap
use keySet iterator where entrySet iterator should be used.
{code}
for (Long cuboid : cuboidAggCostMapCopy.keySet()) {
if (cuboidAggCostMapCopy.get(cuboid) <
processCuboidAggCostMap.get(cuboid)) {
{code}
CubeCodeSystem.newMetricsAggregators has similar issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3578) Do not synchronize on the intrinsic locks of high-level concurrency objects
Ted Yu created KYLIN-3578:
-
Summary: Do not synchronize on the intrinsic locks of high-level
concurrency objects
Key: KYLIN-3578
URL: https://issues.apache.org/jira/browse/KYLIN-3578
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
>From
>core-common/src/main/java/org/apache/kylin/common/util/MemoryBudgetController.java
> :
{code}
private final ReentrantLock lock = new ReentrantLock();
...
synchronized (lock) {
{code}
See the following for why such practice is to be avoided:
https://wiki.sei.cmu.edu/confluence/display/java/LCK03-J.+Do+not+synchronize+on+the+intrinsic+locks+of+high-level+concurrency+objects
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3578) Do not synchronize on the intrinsic locks of high-level concurrency objects
Ted Yu created KYLIN-3578:
-
Summary: Do not synchronize on the intrinsic locks of high-level
concurrency objects
Key: KYLIN-3578
URL: https://issues.apache.org/jira/browse/KYLIN-3578
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
>From
>core-common/src/main/java/org/apache/kylin/common/util/MemoryBudgetController.java
> :
{code}
private final ReentrantLock lock = new ReentrantLock();
...
synchronized (lock) {
{code}
See the following for why such practice is to be avoided:
https://wiki.sei.cmu.edu/confluence/display/java/LCK03-J.+Do+not+synchronize+on+the+intrinsic+locks+of+high-level+concurrency+objects
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Attachment: 21221.v9.txt
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Attachments: 21221.v7.txt, 21221.v8.txt, 21221.v9.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (FLINK-8037) Missing cast in integer arithmetic in TransactionalIdsGenerator#generateIdsToAbort
[
https://issues.apache.org/jira/browse/FLINK-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-8037:
--
Comment: was deleted
(was: Please rebase PR.)
> Missing cast in integer arithmetic in
> TransactionalIdsGenerator#generateIdsToAbort
> --
>
> Key: FLINK-8037
> URL: https://issues.apache.org/jira/browse/FLINK-8037
> Project: Flink
> Issue Type: Bug
> Reporter: Ted Yu
>Assignee: Greg Hogan
>Priority: Minor
> Labels: kafka, kafka-connect
>
> {code}
> public Set generateIdsToAbort() {
> Set idsToAbort = new HashSet<>();
> for (int i = 0; i < safeScaleDownFactor; i++) {
> idsToAbort.addAll(generateIdsToUse(i * poolSize *
> totalNumberOfSubtasks));
> {code}
> The operands are integers where generateIdsToUse() expects long parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Assignee: Ted Yu
Status: Patch Available (was: Open)
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
> Assignee: Ted Yu
>Priority: Minor
> Attachments: 21221.v7.txt, 21221.v8.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Attachment: 21221.v8.txt
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
>Priority: Minor
> Attachments: 21221.v7.txt, 21221.v8.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625060#comment-16625060
]
Ted Yu commented on HBASE-21221:
Patch v8 doesn't wait for the captured exception to be non-null.
Instead, check captured exception toward the end of the test.
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
>Priority: Minor
> Attachments: 21221.v7.txt, 21221.v8.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624975#comment-16624975
]
Ted Yu commented on HBASE-21221:
By subclassing MultiRowMutationEndpoint, I was able to capture the exception
encountered by mutateRows.
However, the testMultiRowMutations tears down before the captured exception can
be examined ...
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
>Priority: Minor
> Attachments: 21221.v7.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
[
https://issues.apache.org/jira/browse/HBASE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21221:
---
Attachment: 21221.v7.txt
> Ineffective assertion in TestFromClientSide3#testMultiRowMutations
> --
>
> Key: HBASE-21221
> URL: https://issues.apache.org/jira/browse/HBASE-21221
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
>Priority: Minor
> Attachments: 21221.v7.txt
>
>
> Observed the following in
> org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
> {code}
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
> 089bdfa75f44d88e596479038a6da18b
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
> at
> org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
> at
> org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
> ...
> Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
> should fail because the target lock is blocked by previous put
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> {code}
> Here is related code:
> {code}
> cpService.execute(() -> {
> ...
> if (!threw) {
> // Can't call fail() earlier because the catch would eat it.
> fail("This cp should fail because the target lock is blocked by
> previous put");
> }
> {code}
> Since the fail() call is executed by the cpService, the assertion had no
> bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9824) Support IPv6 literal
[ https://issues.apache.org/jira/browse/FLINK-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-9824: -- Description: Currently we use colon as separator when parsing host and port. We should support the usage of IPv6 literals in parsing. was: Currently we use colon as separator when parsing host and port. We should support the usage of IPv6 literals in parsing. > Support IPv6 literal > > > Key: FLINK-9824 > URL: https://issues.apache.org/jira/browse/FLINK-9824 > Project: Flink > Issue Type: Bug > Components: Network > Reporter: Ted Yu >Assignee: vinoyang >Priority: Minor > > Currently we use colon as separator when parsing host and port. > We should support the usage of IPv6 literals in parsing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20654:
---
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: 1.png, HBASE-20654-1.patch, HBASE-20654-2.patch,
> HBASE-20654-3.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21141) Enable MOB in backup / restore test involving incremental backup
[ https://issues.apache.org/jira/browse/HBASE-21141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21141: --- Labels: mob (was: ) > Enable MOB in backup / restore test involving incremental backup > > > Key: HBASE-21141 > URL: https://issues.apache.org/jira/browse/HBASE-21141 > Project: HBase > Issue Type: Test > Components: backuprestore > Reporter: Ted Yu >Priority: Major > Labels: mob > > Currently we only have one test (TestRemoteBackup) where MOB feature is > enabled. The test only performs full backup. > This issue is to enable MOB in backup / restore test(s) involving incremental > backup. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
Ted Yu created HBASE-21221:
--
Summary: Ineffective assertion in
TestFromClientSide3#testMultiRowMutations
Key: HBASE-21221
URL: https://issues.apache.org/jira/browse/HBASE-21221
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Observed the following in
org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
{code}
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
089bdfa75f44d88e596479038a6da18b
at
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
at
org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
at
org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
at
org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
at
org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
...
Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
should fail because the target lock is blocked by previous put
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
Here is related code:
{code}
cpService.execute(() -> {
...
if (!threw) {
// Can't call fail() earlier because the catch would eat it.
fail("This cp should fail because the target lock is blocked by
previous put");
}
{code}
Since the fail() call is executed by the cpService, the assertion had no
bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21221) Ineffective assertion in TestFromClientSide3#testMultiRowMutations
Ted Yu created HBASE-21221:
--
Summary: Ineffective assertion in
TestFromClientSide3#testMultiRowMutations
Key: HBASE-21221
URL: https://issues.apache.org/jira/browse/HBASE-21221
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Observed the following in
org.apache.hadoop.hbase.util.TestFromClientSide3WoUnsafe-output.txt :
{code}
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
java.io.IOException: Timed out waiting for lock for row: ROW-1 in region
089bdfa75f44d88e596479038a6da18b
at
org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5816)
at
org.apache.hadoop.hbase.regionserver.HRegion$4.lockRowsAndBuildMiniBatch(HRegion.java:7432)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4008)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3982)
at
org.apache.hadoop.hbase.regionserver.HRegion.mutateRowsWithLocks(HRegion.java:7424)
at
org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint.mutateRows(MultiRowMutationEndpoint.java:116)
at
org.apache.hadoop.hbase.protobuf.generated.MultiRowMutationProtos$MultiRowMutationService.callMethod(MultiRowMutationProtos.java:2266)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8182)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2481)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2463)
...
Exception in thread "pool-678-thread-1" java.lang.AssertionError: This cp
should fail because the target lock is blocked by previous put
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hbase.client.TestFromClientSide3.lambda$testMultiRowMutations$7(TestFromClientSide3.java:861)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
{code}
Here is related code:
{code}
cpService.execute(() -> {
...
if (!threw) {
// Can't call fail() earlier because the catch would eat it.
fail("This cp should fail because the target lock is blocked by
previous put");
}
{code}
Since the fail() call is executed by the cpService, the assertion had no
bearing on the outcome of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-8554) Upgrade AWS SDK
[
https://issues.apache.org/jira/browse/FLINK-8554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-8554:
--
Description:
AWS SDK 1.11.271 fixes a lot of bugs.
One of which would exhibit the following:
{code}
Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
AWS SDK 1.11.375 has been released.
https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/
was:
AWS SDK 1.11.271 fixes a lot of bugs.
One of which would exhibit the following:
{code}
Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
> Upgrade AWS SDK
> ---
>
> Key: FLINK-8554
> URL: https://issues.apache.org/jira/browse/FLINK-8554
> Project: Flink
> Issue Type: Improvement
> Components: Build System
> Reporter: Ted Yu
>Assignee: vinoyang
>Priority: Minor
>
> AWS SDK 1.11.271 fixes a lot of bugs.
> One of which would exhibit the following:
> {code}
> Caused by: java.lang.NullPointerException
> at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
> at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
> at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
> AWS SDK 1.11.375 has been released.
> https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6904) DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate is flaky
[
https://issues.apache.org/jira/browse/KAFKA-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607291#comment-16607291
]
Ted Yu edited comment on KAFKA-6904 at 9/22/18 7:06 PM:
Haven't seen this failure lately .
was (Author: yuzhih...@gmail.com):
Haven't seen this failure lately.
> DynamicBrokerReconfigurationTest#testAdvertisedListenerUpdate is flaky
> --
>
> Key: KAFKA-6904
> URL: https://issues.apache.org/jira/browse/KAFKA-6904
> Project: Kafka
> Issue Type: Test
> Components: unit tests
> Reporter: Ted Yu
>Priority: Major
>
> From
> https://builds.apache.org/job/kafka-pr-jdk10-scala2.12/820/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/testAdvertisedListenerUpdate/
> :
> {code}
> kafka.server.DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate
> Failing for the past 1 build (Since Failed#820 )
> Took 21 sec.
> Error Message
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
> not the leader for that topic-partition.
> Stacktrace
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
> not the leader for that topic-partition.
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:77)
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
> at
> kafka.server.DynamicBrokerReconfigurationTest.$anonfun$verifyProduceConsume$3(DynamicBrokerReconfigurationTest.scala:996)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
> at scala.collection.Iterator.foreach(Iterator.scala:944)
> at scala.collection.Iterator.foreach$(Iterator.scala:944)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
> at scala.collection.IterableLike.foreach(IterableLike.scala:71)
> at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at scala.collection.TraversableLike.map(TraversableLike.scala:234)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at
> kafka.server.DynamicBrokerReconfigurationTest.verifyProduceConsume(DynamicBrokerReconfigurationTest.scala:996)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate(DynamicBrokerReconfigurationTest.scala:742)
> {code}
> The above happened with jdk 10.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3431) Avoid FileInputStream/FileOutputStream
[
https://issues.apache.org/jira/browse/KYLIN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624744#comment-16624744
]
Ted Yu commented on KYLIN-3431:
---
Thanks, Long.
> Avoid FileInputStream/FileOutputStream
> --
>
> Key: KYLIN-3431
> URL: https://issues.apache.org/jira/browse/KYLIN-3431
> Project: Kylin
> Issue Type: Improvement
> Reporter: Ted Yu
>Priority: Major
> Fix For: Backlog
>
>
> They rely on finalizers (before Java 11), which create unnecessary GC load.
> The alternatives, {{Files.newInputStream}}, are as easy to use and don't have
> this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624443#comment-16624443
]
Ted Yu commented on KYLIN-3417:
---
For phase I, we don't need to use Optimistic read lock
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
> Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (GORA-544) Upgrade zookeeper to 3.4.13
Ted Yu created GORA-544: --- Summary: Upgrade zookeeper to 3.4.13 Key: GORA-544 URL: https://issues.apache.org/jira/browse/GORA-544 Project: Apache Gora Issue Type: Task Reporter: Ted Yu zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (GORA-542) Upgrade to JUnit 5
[ https://issues.apache.org/jira/browse/GORA-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated GORA-542: Description: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 was: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 > Upgrade to JUnit 5 > -- > > Key: GORA-542 > URL: https://issues.apache.org/jira/browse/GORA-542 > Project: Apache Gora > Issue Type: Task > Reporter: Ted Yu >Priority: Major > > JUnit 5 brings multiple useful features so tests are easier to read and write. > We can bump up the dependency version and create new tests with JUnit 5 > features. > Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests > https://twitter.com/nipafx/status/1027095088059559936 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10391) MillisOfDay is used in place of instant for LocalTime ctor in AvroKryoSerializerUtils
Ted Yu created FLINK-10391:
--
Summary: MillisOfDay is used in place of instant for LocalTime
ctor in AvroKryoSerializerUtils
Key: FLINK-10391
URL: https://issues.apache.org/jira/browse/FLINK-10391
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
>From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value
>from LocalTime.
For read method:
{code}
final int time = input.readInt(true);
return new LocalTime(time,
ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
{code}
It seems
http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3447) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/KYLIN-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3447: -- Description: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: KYLIN-3447 > URL: https://issues.apache.org/jira/browse/KYLIN-3447 > Project: Kylin > Issue Type: Improvement > Reporter: Ted Yu >Priority: Major > > zookeeper 3.4.13 is being released with the following fixes: > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) > environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes . (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes.) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task > Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3519) Upgrade Jacoco version to 0.8.2
[ https://issues.apache.org/jira/browse/KYLIN-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3519: -- Description: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. was: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. > Upgrade Jacoco version to 0.8.2 > --- > > Key: KYLIN-3519 > URL: https://issues.apache.org/jira/browse/KYLIN-3519 > Project: Kylin > Issue Type: Improvement > Reporter: Ted Yu >Priority: Minor > > Jacoco 0.8.2 adds Java 11 support: >https://github.com/jacoco/jacoco/releases/tag/v0.8.2 > Java 11 RC1 is out. > We should consider upgrading Jacoco. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3559) Use Splitter for splitting String
[ https://issues.apache.org/jira/browse/KYLIN-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3559: -- Description: See http://errorprone.info/bugpattern/StringSplitter for why Splitter is preferred . (was: See http://errorprone.info/bugpattern/StringSplitter for why Splitter is preferred.) > Use Splitter for splitting String > - > > Key: KYLIN-3559 > URL: https://issues.apache.org/jira/browse/KYLIN-3559 > Project: Kylin > Issue Type: Task > Reporter: Ted Yu >Priority: Major > Fix For: v2.6.0 > > > See http://errorprone.info/bugpattern/StringSplitter for why Splitter is > preferred . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20578) Support region server group in target cluster
[ https://issues.apache.org/jira/browse/HBASE-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605078#comment-16605078 ] Ted Yu edited comment on HBASE-20578 at 9/21/18 8:54 PM: - Ping [~albertlee...@gmail.com] Do you have any further question ? was (Author: yuzhih...@gmail.com): Ping [~albertlee...@gmail.com] > Support region server group in target cluster > - > > Key: HBASE-20578 > URL: https://issues.apache.org/jira/browse/HBASE-20578 > Project: HBase > Issue Type: Improvement > Components: Replication, rsgroup > Reporter: Ted Yu >Assignee: Albert Lee >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20578-001.patch > > > When source tables belong to non-default region server group(s) and there are > region server group counterpart in the target cluster, we should support > replicating to target cluster using the region server group mapping. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10389) TaskManagerServicesConfiguration ctor contains self assignment
Ted Yu created FLINK-10389:
--
Summary: TaskManagerServicesConfiguration ctor contains self
assignment
Key: FLINK-10389
URL: https://issues.apache.org/jira/browse/FLINK-10389
Project: Flink
Issue Type: Task
Reporter: Ted Yu
TaskManagerServicesConfiguration has:
{code}
this.systemResourceMetricsEnabled = systemResourceMetricsEnabled;
{code}
There is no systemResourceMetricsEnabled parameter to the ctor.
This was reported by findbugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10389) TaskManagerServicesConfiguration ctor contains self assignment
Ted Yu created FLINK-10389:
--
Summary: TaskManagerServicesConfiguration ctor contains self
assignment
Key: FLINK-10389
URL: https://issues.apache.org/jira/browse/FLINK-10389
Project: Flink
Issue Type: Task
Reporter: Ted Yu
TaskManagerServicesConfiguration has:
{code}
this.systemResourceMetricsEnabled = systemResourceMetricsEnabled;
{code}
There is no systemResourceMetricsEnabled parameter to the ctor.
This was reported by findbugs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10388) RestClientTest sometimes fails with AssertionError
Ted Yu created FLINK-10388:
--
Summary: RestClientTest sometimes fails with AssertionError
Key: FLINK-10388
URL: https://issues.apache.org/jira/browse/FLINK-10388
Project: Flink
Issue Type: Test
Reporter: Ted Yu
Running the test on Linux I got:
{code}
testConnectionTimeout(org.apache.flink.runtime.rest.RestClientTest) Time
elapsed: 1.918 sec <<< FAILURE!
java.lang.AssertionError:
Expected: an instance of
org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
but:
is a
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedSocketException
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at
org.apache.flink.runtime.rest.RestClientTest.testConnectionTimeout(RestClientTest.java:69)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10388) RestClientTest sometimes fails with AssertionError
Ted Yu created FLINK-10388:
--
Summary: RestClientTest sometimes fails with AssertionError
Key: FLINK-10388
URL: https://issues.apache.org/jira/browse/FLINK-10388
Project: Flink
Issue Type: Test
Reporter: Ted Yu
Running the test on Linux I got:
{code}
testConnectionTimeout(org.apache.flink.runtime.rest.RestClientTest) Time
elapsed: 1.918 sec <<< FAILURE!
java.lang.AssertionError:
Expected: an instance of
org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
but:
is a
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedSocketException
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at
org.apache.flink.runtime.rest.RestClientTest.testConnectionTimeout(RestClientTest.java:69)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3575) Unclosed Connection in DriverTest
Ted Yu created KYLIN-3575:
-
Summary: Unclosed Connection in DriverTest
Key: KYLIN-3575
URL: https://issues.apache.org/jira/browse/KYLIN-3575
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In testCalciteProps :
{code}
KylinConnection conn = (KylinConnection)
driver.connect("jdbc:kylin:test_url/test_db", props);
{code}
The connection should be closed upon return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3575) Unclosed Connection in DriverTest
Ted Yu created KYLIN-3575:
-
Summary: Unclosed Connection in DriverTest
Key: KYLIN-3575
URL: https://issues.apache.org/jira/browse/KYLIN-3575
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In testCalciteProps :
{code}
KylinConnection conn = (KylinConnection)
driver.connect("jdbc:kylin:test_url/test_db", props);
{code}
The connection should be closed upon return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3574) Unclosed lookup table in DictionaryGeneratorCLI#processSegment
Ted Yu created KYLIN-3574:
-
Summary: Unclosed lookup table in
DictionaryGeneratorCLI#processSegment
Key: KYLIN-3574
URL: https://issues.apache.org/jira/browse/KYLIN-3574
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
JoinDesc join =
cubeSeg.getModel().getJoinsTree().getJoinByPKSide(lookup);
cubeMgr.getLookupTable(cubeSeg, join);
{code}
The returned lookup table should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3574) Unclosed lookup table in DictionaryGeneratorCLI#processSegment
Ted Yu created KYLIN-3574:
-
Summary: Unclosed lookup table in
DictionaryGeneratorCLI#processSegment
Key: KYLIN-3574
URL: https://issues.apache.org/jira/browse/KYLIN-3574
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
JoinDesc join =
cubeSeg.getModel().getJoinsTree().getJoinByPKSide(lookup);
cubeMgr.getLookupTable(cubeSeg, join);
{code}
The returned lookup table should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AMBARI-24674) Upgrade checkstyle version to 8.9
Ted Yu created AMBARI-24674: --- Summary: Upgrade checkstyle version to 8.9 Key: AMBARI-24674 URL: https://issues.apache.org/jira/browse/AMBARI-24674 Project: Ambari Issue Type: Task Reporter: Ted Yu Currently we use checkstyle 6.19 which is very old. We should upgrade to 8.9 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Can MOB be enabled for existing table?
bq. region server crashes after ttl archiver runs This is the first time I hear of such problem. Can you give us more information (stack trace, server log snippet for that period) ? Thanks On Thu, Sep 20, 2018 at 11:14 PM meirav.malka wrote: > Hi,Since you probably don't have enough space for backup for such a large > table, test it on a small one and rowcount before and after.We work > extensively with mob and there are several issues with it that we had to > work around.1.Mob and ttl - region server crashes after ttl archiver > runs.2.Mob and snapshot - didn't work in cdh 5.7.13.Mob compaction- > currently performed only by the master.if your flushes aren't big enough , > compaction will never end.4.Deletes - major mob compaction is the only one > doing mob delete,it is also done by the master, on the entire data( it is > not scheduled )If you have deletes, the will never actually be deleted. > Regards,Meirav > Sent from my Samsung Galaxy smartphone. > Original message From: Jean-Marc Spaggiari < > jean-m...@spaggiari.org> Date: 9/21/18 01:09 (GMT+02:00) To: user < > user@hbase.apache.org> Subject: Re: Can MOB be enabled for existing > table? > Hi Andrea, > > I will say that the easiest way to know is to try it... I'm guessing that > it should work. When HBase will compact the file, it will figure that some > fields are bigger than the configures MOB threshold and will move them > under the MOB umbrella. But this has to be tested. Let us know. > > Thanks, > > JMS > > Le jeu. 20 sept. 2018 à 08:26, Andrea Fonti a > écrit : > > > Hi, > > > > I’m currently running HBASE on a Cloudera CDH 5.8.4 distribution > > (hbase-1.2.0+cdh5.8.5+263), is it possible to enable MOB for a column of > an > > existing multi terabyte table? > > > > If the answer is affirmative what will hbase do with the existing data? > > Will it rewrite with MOB files the next major compaction? > > > > Thank you > > > > Andrea >
[jira] [Commented] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure
[
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623691#comment-16623691
]
Ted Yu commented on HBASE-21149:
>From
>https://builds.apache.org/job/HBase%20Nightly/job/master/503//testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/health_checks___yetus_jdk8_hadoop3_checks___TestIncBackupDeleteTable/
> :
{code}
java.io.IOException: Failed copy from
hdfs://localhost:40355/user/jenkins/test-data/6d0386bb-d9d8-ae4e-e2bf-c423554ea898/data/default/test-1537528002870/711c2cc8826d7170be6051afbe0b5ca0/f/1afeb42e839a40afa6389ac80c82e1ed_SeqId_205_,hdfs://localhost:40355/user/jenkins/test-data/6d0386bb-d9d8-ae4e-e2bf-c423554ea898/data/default/test-1537528002870/711c2cc8826d7170be6051afbe0b5ca0/f/2087ea2ec81a449ea5d35c405767f970_SeqId_205_
to hdfs://localhost:40355/backupUT/backup_1537528036126
{code}
There were some output truncated.
I searched for 2087ea2ec81a449ea5d35c405767f970_SeqId_205_ in the output but
didn't find much clue.
> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
> Issue Type: Test
> Components: backuprestore
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
> :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test]
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
> at
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6303) Potential lack of synchronization in NioEchoServer#AcceptorThread
[
https://issues.apache.org/jira/browse/KAFKA-6303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333251#comment-16333251
]
Ted Yu edited comment on KAFKA-6303 at 9/21/18 1:50 AM:
+1 from me .
was (Author: yuzhih...@gmail.com):
+1 from me.
> Potential lack of synchronization in NioEchoServer#AcceptorThread
> -
>
> Key: KAFKA-6303
> URL: https://issues.apache.org/jira/browse/KAFKA-6303
> Project: Kafka
> Issue Type: Bug
> Components: network
> Reporter: Ted Yu
>Assignee: siva santhalingam
>Priority: Minor
>
> In the run() method:
> {code}
> SocketChannel socketChannel =
> ((ServerSocketChannel) key.channel()).accept();
> socketChannel.configureBlocking(false);
> newChannels.add(socketChannel);
> {code}
> Modification to newChannels should be protected by synchronized block.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-16458) Shorten backup / restore test execution time
[
https://issues.apache.org/jira/browse/HBASE-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16458:
---
Fix Version/s: 3.0.0
> Shorten backup / restore test execution time
>
>
> Key: HBASE-16458
> URL: https://issues.apache.org/jira/browse/HBASE-16458
> Project: HBase
> Issue Type: Test
> Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Labels: backup
> Fix For: 3.0.0
>
> Attachments: 16458-v1.patch, 16458.HBASE-7912.v3.txt,
> 16458.HBASE-7912.v4.txt, 16458.v1.txt, 16458.v2.txt, 16458.v2.txt,
> 16458.v3.txt, 16458.v4.txt, 16458.v5.txt, HBASE-16458-v1.patch,
> HBASE-16458-v2.patch
>
>
> Below was timing information for all the backup / restore tests (today's
> result):
> {code}
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackup
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 576.273 sec -
> in org.apache.hadoop.hbase.backup.TestIncrementalBackup
> Running org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.67 sec -
> in org.apache.hadoop.hbase.backup.TestBackupBoundaryTests
> Running org.apache.hadoop.hbase.backup.TestBackupStatusProgress
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.34 sec -
> in org.apache.hadoop.hbase.backup.TestBackupStatusProgress
> Running org.apache.hadoop.hbase.backup.TestBackupAdmin
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 490.251 sec -
> in org.apache.hadoop.hbase.backup.TestBackupAdmin
> Running org.apache.hadoop.hbase.backup.TestHFileArchiving
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.323 sec -
> in org.apache.hadoop.hbase.backup.TestHFileArchiving
> Running org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.492 sec -
> in org.apache.hadoop.hbase.backup.TestSystemTableSnapshot
> Running org.apache.hadoop.hbase.backup.TestBackupDescribe
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.758 sec -
> in org.apache.hadoop.hbase.backup.TestBackupDescribe
> Running org.apache.hadoop.hbase.backup.TestBackupLogCleaner
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.187 sec -
> in org.apache.hadoop.hbase.backup.TestBackupLogCleaner
> Running org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 330.539 sec -
> in org.apache.hadoop.hbase.backup.TestIncrementalBackupNoDataLoss
> Running org.apache.hadoop.hbase.backup.TestRemoteBackup
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.371 sec -
> in org.apache.hadoop.hbase.backup.TestRemoteBackup
> Running org.apache.hadoop.hbase.backup.TestBackupSystemTable
> Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.893 sec -
> in org.apache.hadoop.hbase.backup.TestBackupSystemTable
> Running org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.779 sec -
> in org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests
> Running org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.815 sec -
> in org.apache.hadoop.hbase.backup.TestFullBackupSetRestoreSet
> Running org.apache.hadoop.hbase.backup.TestBackupShowHistory
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.517 sec -
> in org.apache.hadoop.hbase.backup.TestBackupShowHistory
> Running org.apache.hadoop.hbase.backup.TestRemoteRestore
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.799 sec -
> in org.apache.hadoop.hbase.backup.TestRemoteRestore
> Running org.apache.hadoop.hbase.backup.TestFullRestore
> Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 317.711 sec
> - in org.apache.hadoop.hbase.backup.TestFullRestore
> Running org.apache.hadoop.hbase.backup.TestFullBackupSet
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.045 sec -
> in org.apache.hadoop.hbase.backup.TestFullBackupSet
> Running org.apache.hadoop.hbase.backup.TestBackupDelete
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 86.214 sec -
> in org.apache.hadoop.hbase.backup.TestBackupDelete
> Running org.apache.hadoop.hbase.backup.TestBackupDeleteRestore
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.631 sec -
> in org.apache.hadoop.hbase.backup.TestBackup
[jira] [Created] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
Ted Yu created HBASE-21216:
--
Summary: TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
Key: HBASE-21216
URL: https://issues.apache.org/jira/browse/HBASE-21216
Project: HBase
Issue Type: Test
Reporter: Ted Yu
>From
>https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
{code}
java.lang.AssertionError: Archived hfiles [] and table hfiles
[9ca09392705f425f9c916beedc10d63c] is missing snapshot
file:6739a09747e54189a4112a6d8f37e894
at
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
{code}
The file appeared in archive dir before hfile cleaners were run:
{code}
2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-archive/
2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
|data/
2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
|---default/
2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
|--test/
2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-1237d57b63a7bdf067a930441a02514a/
2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
|recovered.edits/
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---4.seqid
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-29e1700e09b51223ad2f5811105a4d51/
2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
|fam/
2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---2c66a18f6c1a4074b84ffbb3245268c4
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---45bb396c6a5e49629e45a4d56f1e9b14
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---6739a09747e54189a4112a6d8f37e894
{code}
However, the archive dir became empty after hfile cleaners were run:
{code}
2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-archive/
2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-corrupt/
{code}
Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
Ted Yu created HBASE-21216:
--
Summary: TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
Key: HBASE-21216
URL: https://issues.apache.org/jira/browse/HBASE-21216
Project: HBase
Issue Type: Test
Reporter: Ted Yu
>From
>https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
{code}
java.lang.AssertionError: Archived hfiles [] and table hfiles
[9ca09392705f425f9c916beedc10d63c] is missing snapshot
file:6739a09747e54189a4112a6d8f37e894
at
org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
{code}
The file appeared in archive dir before hfile cleaners were run:
{code}
2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-archive/
2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
|data/
2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
|---default/
2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
|--test/
2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-1237d57b63a7bdf067a930441a02514a/
2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
|recovered.edits/
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---4.seqid
2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-29e1700e09b51223ad2f5811105a4d51/
2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
|fam/
2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---2c66a18f6c1a4074b84ffbb3245268c4
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---45bb396c6a5e49629e45a4d56f1e9b14
2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
|---6739a09747e54189a4112a6d8f37e894
{code}
However, the archive dir became empty after hfile cleaners were run:
{code}
2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-archive/
2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
|-corrupt/
{code}
Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-7795) Utilize error-prone to discover common coding mistakes
[
https://issues.apache.org/jira/browse/FLINK-7795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345955#comment-16345955
]
Ted Yu edited comment on FLINK-7795 at 9/20/18 4:30 PM:
error-prone has JDK 8 dependency .
was (Author: yuzhih...@gmail.com):
error-prone has JDK 8 dependency.
> Utilize error-prone to discover common coding mistakes
> --
>
> Key: FLINK-7795
> URL: https://issues.apache.org/jira/browse/FLINK-7795
> Project: Flink
> Issue Type: Improvement
> Components: Build System
> Reporter: Ted Yu
>Priority: Major
>
> http://errorprone.info/ is a tool which detects common coding mistakes.
> We should incorporate into Flink build process.
> Here are the dependencies:
> {code}
>
> com.google.errorprone
> error_prone_annotation
> ${error-prone.version}
> provided
>
>
>
> com.google.auto.service
> auto-service
> 1.0-rc3
> true
>
>
> com.google.errorprone
> error_prone_check_api
> ${error-prone.version}
> provided
>
>
> com.google.code.findbugs
> jsr305
>
>
>
>
> com.google.errorprone
> javac
> 9-dev-r4023-3
> provided
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (FLINK-6105) Properly handle InterruptedException in HadoopInputFormatBase
[
https://issues.apache.org/jira/browse/FLINK-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307281#comment-16307281
]
Ted Yu edited comment on FLINK-6105 at 9/20/18 4:29 PM:
In
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
:
{code}
try {
Thread.sleep(500);
} catch (InterruptedException e1) {
// ignore it
}
{code}
Interrupt status should be restored, or throw InterruptedIOException.
was (Author: yuzhih...@gmail.com):
In
flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/RollingSink.java
:
{code}
try {
Thread.sleep(500);
} catch (InterruptedException e1) {
// ignore it
}
{code}
Interrupt status should be restored, or throw InterruptedIOException .
> Properly handle InterruptedException in HadoopInputFormatBase
> -
>
> Key: FLINK-6105
> URL: https://issues.apache.org/jira/browse/FLINK-6105
> Project: Flink
> Issue Type: Bug
> Components: DataStream API
> Reporter: Ted Yu
>Assignee: zhangminglei
>Priority: Major
>
> When catching InterruptedException, we should throw InterruptedIOException
> instead of IOException.
> The following example is from HadoopInputFormatBase :
> {code}
> try {
> splits = this.mapreduceInputFormat.getSplits(jobContext);
> } catch (InterruptedException e) {
> throw new IOException("Could not get Splits.", e);
> }
> {code}
> There may be other places where IOE is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (GORA-543) Upgrade Apache parent POM to version 21
Ted Yu created GORA-543: --- Summary: Upgrade Apache parent POM to version 21 Key: GORA-543 URL: https://issues.apache.org/jira/browse/GORA-543 Project: Apache Gora Issue Type: Task Reporter: Ted Yu Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3573) Upgrade Apache parent POM to version 21
Ted Yu created KYLIN-3573: - Summary: Upgrade Apache parent POM to version 21 Key: KYLIN-3573 URL: https://issues.apache.org/jira/browse/KYLIN-3573 Project: Kylin Issue Type: Task Reporter: Ted Yu Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3573) Upgrade Apache parent POM to version 21
Ted Yu created KYLIN-3573: - Summary: Upgrade Apache parent POM to version 21 Key: KYLIN-3573 URL: https://issues.apache.org/jira/browse/KYLIN-3573 Project: Kylin Issue Type: Task Reporter: Ted Yu Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21204) NPE when scan raw DELETE_FAMILY_VERSION and codec is not set
[
https://issues.apache.org/jira/browse/HBASE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21204:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: (was: 2.0.0)
3.0.0
Status: Resolved (was: Patch Available)
Thanks for the patch, jingyun
> NPE when scan raw DELETE_FAMILY_VERSION and codec is not set
>
>
> Key: HBASE-21204
> URL: https://issues.apache.org/jira/browse/HBASE-21204
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.0, 2.0.0, 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.0
>
> Attachments: HBASE-21204.branch-2.001.patch,
> HBASE-21204.branch-2.002.patch, HBASE-21204.master.001.patch,
> HBASE-21204.master.002.patch, HBASE-21204.master.003.patch
>
>
> There are 7 types of our Cell,
> Minimum((byte)0),
> Put((byte)4),
> Delete((byte)8),
> DeleteFamilyVersion((byte)10),
> DeleteColumn((byte)12),
> DeleteFamily((byte)14),
> Maximum((byte)255);
> But there are only 6 types of our CellType protobuf definition:
> enum CellType {
> MINIMUM = 0;
> PUT = 4;
> DELETE = 8;
> DELETE_FAMILY_VERSION = 10;
> DELETE_COLUMN = 12;
> DELETE_FAMILY = 14;
> MAXIMUM = 255;
> }
> Thus if we scan raw data which is DELETE_FAMILY_VERSION，it will throw NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9924) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/FLINK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-9924: -- Description: zookeeper 3.4.13 is being released. ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released. ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: FLINK-9924 > URL: https://issues.apache.org/jira/browse/FLINK-9924 > Project: Flink > Issue Type: Task > Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > > zookeeper 3.4.13 is being released. > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3572) Upgrade commons-dbcp to latest commons-dbcp2
Ted Yu created KYLIN-3572: - Summary: Upgrade commons-dbcp to latest commons-dbcp2 Key: KYLIN-3572 URL: https://issues.apache.org/jira/browse/KYLIN-3572 Project: Kylin Issue Type: Task Reporter: Ted Yu There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3572) Upgrade commons-dbcp to latest commons-dbcp2
Ted Yu created KYLIN-3572: - Summary: Upgrade commons-dbcp to latest commons-dbcp2 Key: KYLIN-3572 URL: https://issues.apache.org/jira/browse/KYLIN-3572 Project: Kylin Issue Type: Task Reporter: Ted Yu There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621260#comment-16621260
]
Ted Yu commented on HBASE-20952:
Thanks for the patience.
I have modified the google doc link above for a high level design.
Thanks to Josh and Sergey for internal reviews.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AMBARI-18952) Register BackupObserver and BackupHFileCleaner
[ https://issues.apache.org/jira/browse/AMBARI-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884466#comment-15884466 ] Ted Yu edited comment on AMBARI-18952 at 9/19/18 5:40 PM: -- Note: There is cost when BackupObserver is registered - it would poll backup table (at the end of bulk load per region) for whether the underlying table has gone thru full backup. was (Author: yuzhih...@gmail.com): Note: There is cost when BackupObserver is registered - it would poll backup table (at the end of bulk load per region) for whether the underlying table has gone thru full backup . > Register BackupObserver and BackupHFileCleaner > -- > > Key: AMBARI-18952 > URL: https://issues.apache.org/jira/browse/AMBARI-18952 > Project: Ambari > Issue Type: Improvement > Reporter: Ted Yu >Priority: Major > > Over in HBASE-14417, two new classes are added. > org.apache.hadoop.hbase.backup.BackupHFileCleaner should be registered > through hbase.master.hfilecleaner.plugins . It is responsible for keeping > bulk loaded hfiles so that incremental backup can pick them up. > org.apache.hadoop.hbase.backup.BackupObserver should be registered through > hbase.coprocessor.region.classes > It is notified when bulk load completes and writes records into hbase:backup > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AMBARI-18528) Need a way to escape config values that contain $
[
https://issues.apache.org/jira/browse/AMBARI-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603928#comment-15603928
]
Ted Yu edited comment on AMBARI-18528 at 9/19/18 5:40 PM:
--
Escaping the dollar sign automatically is a desirable solution .
was (Author: yuzhih...@gmail.com):
Escaping the dollar sign automatically is a desirable solution.
> Need a way to escape config values that contain $
> -
>
> Key: AMBARI-18528
> URL: https://issues.apache.org/jira/browse/AMBARI-18528
> Project: Ambari
> Issue Type: Bug
> Reporter: Ted Yu
>Priority: Major
> Labels: zookeeper
>
> We tried specifying auth_to_local in zookeeper env input box:
> {code}
> export SERVER_JVMFLAGS="$SERVER_JVMFLAGS
> -Dzookeeper.security.auth_to_local=RULE:[2:$1@$0](hb...@c.net)s/.*/hbase/
> -Djava.security.auth.login.config={{zk_server_jaas_file}}"
> {code}
> However, when zookeeper quorum starts, the rule got interrupted with
> zkServer.sh
> We should add the capability of specifying auth_to_local in Ambari UI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-22965) Boolean#getBoolean is used to parse value
[
https://issues.apache.org/jira/browse/AMBARI-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated AMBARI-22965:
Description:
In StormTimelineMetricsReporter.java :
{code}
if (cf.containsKey(SET_INSTANCE_ID_PROPERTY)) {
setInstanceId =
Boolean.getBoolean(cf.get(SET_INSTANCE_ID_PROPERTY).toString());
{code}
According to
https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
{code}
Returns true if and only if the system property named by the argument exists
and is equal to the string "true"
{code}
This was not the intention of the quoted code.
FlumeTimelineMetricsSink.java may have the same issue.
was:
In StormTimelineMetricsReporter.java :
{code}
if (cf.containsKey(SET_INSTANCE_ID_PROPERTY)) {
setInstanceId =
Boolean.getBoolean(cf.get(SET_INSTANCE_ID_PROPERTY).toString());
{code}
According to
https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
{code}
Returns true if and only if the system property named by the argument exists
and is equal to the string "true"
{code}
This was not the intention of the quoted code.
FlumeTimelineMetricsSink.java may have the same issue.
> Boolean#getBoolean is used to parse value
> -
>
> Key: AMBARI-22965
> URL: https://issues.apache.org/jira/browse/AMBARI-22965
> Project: Ambari
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
>
> In StormTimelineMetricsReporter.java :
> {code}
> if (cf.containsKey(SET_INSTANCE_ID_PROPERTY)) {
> setInstanceId =
> Boolean.getBoolean(cf.get(SET_INSTANCE_ID_PROPERTY).toString());
> {code}
> According to
> https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
> {code}
> Returns true if and only if the system property named by the argument exists
> and is equal to the string "true"
> {code}
> This was not the intention of the quoted code.
> FlumeTimelineMetricsSink.java may have the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: from_csv
+1
Original message From: Dongjin Lee Date:
9/19/18 7:20 AM (GMT-08:00) To: dev Subject: Re:
from_csv
Another +1.
I already experienced this case several times.
On Mon, Sep 17, 2018 at 11:03 AM Hyukjin Kwon wrote:
+1 for this idea since text parsing in CSV/JSON is quite common.
One thing is about schema inference likewise with JSON functionality. In case
of JSON, we added schema_of_json for it and same thing should be able to apply
to CSV too.
If we see some more needs for it, we can consider a function like schema_of_csv
as well.
2018년 9월 16일 (일) 오후 4:41, Maxim Gekk 님이 작성:
Hi Reynold,
> i'd make this as consistent as to_json / from_json as possible
Sure, new function from_csv() has the same signature as from_json().
> how would this work in sql? i.e. how would passing options in work?
The options are passed to the function via map, for example:select
from_csv('26/08/2015', 'time Timestamp', map('timestampFormat', 'dd/MM/'))
On Sun, Sep 16, 2018 at 7:01 AM Reynold Xin wrote:
makes sense - i'd make this as consistent as to_json / from_json as possible.
how would this work in sql? i.e. how would passing options in work?
--excuse the brevity and lower case due to wrist injury
On Sat, Sep 15, 2018 at 2:58 AM Maxim Gekk wrote:
Hi All,
I would like to propose new function from_csv() for parsing columns containing
strings in CSV format. Here is my PR: https://github.com/apache/spark/pull/22379
An use case is loading a dataset from an external storage, dbms or systems like
Kafka to where CSV content was dumped as one of columns/fields. Other columns
could contain related information like timestamps, ids, sources of data and
etc. The column with CSV strings can be parsed by existing method csv() of
DataFrameReader but in that case we have to "clean up" dataset and remove other
columns since the csv() method requires Dataset[String]. Joining back result of
parsing and original dataset by positions is expensive and not convenient.
Instead users parse CSV columns by string functions. The approach is usually
error prone especially for quoted values and other special cases.
The proposed in the PR methods should make a better user experience in parsing
CSV-like columns. Please, share your thoughts.
--
Maxim Gekk
Technical Solutions LeadDatabricks inc.maxim.g...@databricks.comdatabricks.com
--
Dongjin Lee
A hitchhiker in the mathematical world.
github: github.com/dongjinleekrlinkedin:
kr.linkedin.com/in/dongjinleekrslideshare: www.slideshare.net/dongjinleekr
[jira] [Commented] (HBASE-21102) ServerCrashProcedure should select target server where no other replicas exist for the current region
[ https://issues.apache.org/jira/browse/HBASE-21102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620116#comment-16620116 ] Ted Yu commented on HBASE-21102: I will commit tomorrow. The failing test is in hbase-rsgroups, a different module. So qa didn't run that test. > ServerCrashProcedure should select target server where no other replicas > exist for the current region > - > > Key: HBASE-21102 > URL: https://issues.apache.org/jira/browse/HBASE-21102 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 3.0.0, 2.2.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: 21102.addendum2.txt, HBASE-21102_1.patch, > HBASE-21102_2.patch, HBASE-21102_3.patch, HBASE-21102_4.patch, > HBASE-21102_addendum.patch, HBASE-21102_addendum.patch, > HBASE-21102_addendum.patch, HBASE-21102_branch-2.1.patch, > HBASE-21102_branch-2.1.patch, HBASE-21102_initial.patch > > > Currently when a server with region replica crashes, when the target server > is created for the replica region assignment there is no guarentee that a > server is selected where there is no other replica for the current region > getting assigned. It so happens that currently we do an assignment randomly > and later the LB comes and identifies these cases and again does MOVE for > such regions. It will be better if we can identify target servers at least > minimally ensuring that replicas are not colocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
