[jira] [Resolved] (PHOENIX-4637) support hbase 1.0
[ https://issues.apache.org/jira/browse/PHOENIX-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva resolved PHOENIX-4637. - Resolution: Invalid > support hbase 1.0 > - > > Key: PHOENIX-4637 > URL: https://issues.apache.org/jira/browse/PHOENIX-4637 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.12.0 >Reporter: cmd >Priority: Major > Fix For: 4.12.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5023) [UMBRELLA] Clean up giant anonymous inner classes
[ https://issues.apache.org/jira/browse/PHOENIX-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-5023: Summary: [UMBRELLA] Clean up giant anonymous inner classes (was: Clean up giant anonymous inner classes) > [UMBRELLA] Clean up giant anonymous inner classes > - > > Key: PHOENIX-5023 > URL: https://issues.apache.org/jira/browse/PHOENIX-5023 > Project: Phoenix > Issue Type: Improvement >Reporter: Geoffrey Jacoby >Priority: Minor > > A frequent anti-pattern in the Phoenix code base is the use of huge anonymous > inner classes, often in the midst of functions that are already very large, > or even the parameter list of constructor calls. (The latter means that we > have single statements that are technically hundreds of lines long!) > This makes the logic difficult to follow and prevents abstraction. Modern > IDEs make changing these to named inner classes quite simple. > This is intended as an umbrella item with subtasks refactoring particular > classes, which can be added to over time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5265) Phoenix Test should use gold files for result comparison instead of using hard-corded comparison.
[
https://issues.apache.org/jira/browse/PHOENIX-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas D'Silva updated PHOENIX-5265:
Labels: hardening (was: )
> Phoenix Test should use gold files for result comparison instead of using
> hard-corded comparison.
> -
>
> Key: PHOENIX-5265
> URL: https://issues.apache.org/jira/browse/PHOENIX-5265
> Project: Phoenix
> Issue Type: Improvement
> Environment: {code:java}
> // code placeholder
> @Test
> public void testWithMultiCF() throws Exception {
> int nRows = 20;
> Connection conn = getConnection(0);
> PreparedStatement stmt;
> conn.createStatement().execute(
> "CREATE TABLE " + fullTableName
> + "(k VARCHAR PRIMARY KEY, a.v INTEGER, b.v INTEGER, c.v
> INTEGER NULL, d.v INTEGER NULL) "
> + tableDDLOptions );
> stmt = conn.prepareStatement("UPSERT INTO " + fullTableName + "
> VALUES(?,?, ?, ?, ?)");
> byte[] val = new byte[250];
> for (int i = 0; i < nRows; i++) {
> stmt.setString(1, Character.toString((char)('a' + i)) +
> Bytes.toString(val));
> stmt.setInt(2, i);
> stmt.setInt(3, i);
> stmt.setInt(4, i);
> stmt.setInt(5, i);
> stmt.executeUpdate();
> }
> conn.commit();
> stmt = conn.prepareStatement("UPSERT INTO " + fullTableName + "(k, c.v,
> d.v) VALUES(?,?,?)");
> for (int i = 0; i < 5; i++) {
> stmt.setString(1, Character.toString((char)('a' + 'z' + i)) +
> Bytes.toString(val));
> stmt.setInt(2, i);
> stmt.setInt(3, i);
> stmt.executeUpdate();
> }
> conn.commit();
> ResultSet rs;
> String actualExplainPlan;
> collectStatistics(conn, fullTableName);
> List keyRanges = getAllSplits(conn, fullTableName);
> assertEquals(26, keyRanges.size());
> rs = conn.createStatement().executeQuery("EXPLAIN SELECT * FROM " +
> fullTableName);
> actualExplainPlan = QueryUtil.getExplainPlan(rs);
> assertEquals(
> "CLIENT 26-CHUNK 25 ROWS " + (columnEncoded ? ( mutable ? "12530"
> : "14190" ) :
> (TransactionFactory.Provider.OMID.name().equals(transactionProvider)) ?
> "25320" : "12420") +
> " BYTES PARALLEL 1-WAY FULL SCAN OVER " +
> physicalTableName,
> actualExplainPlan);
> ConnectionQueryServices services =
> conn.unwrap(PhoenixConnection.class).getQueryServices();
> List regions =
> services.getAllTableRegions(Bytes.toBytes(physicalTableName));
> assertEquals(1, regions.size());
> collectStatistics(conn, fullTableName, Long.toString(1000));
> keyRanges = getAllSplits(conn, fullTableName);
> boolean oneCellPerColFamilyStorageScheme = !mutable && columnEncoded;
> boolean hasShadowCells =
> TransactionFactory.Provider.OMID.name().equals(transactionProvider);
> assertEquals(oneCellPerColFamilyStorageScheme ? 14 : hasShadowCells ? 24
> : 13, keyRanges.size());
> rs = conn
> .createStatement()
> .executeQuery(
> "SELECT
> COLUMN_FAMILY,SUM(GUIDE_POSTS_ROW_COUNT),SUM(GUIDE_POSTS_WIDTH),COUNT(*) from
> \"SYSTEM\".STATS where PHYSICAL_NAME = '"
> + physicalTableName + "' GROUP BY COLUMN_FAMILY
> ORDER BY COLUMN_FAMILY");
> assertTrue(rs.next());
> assertEquals("A", rs.getString(1));
> assertEquals(25, rs.getInt(2));
> assertEquals(columnEncoded ? ( mutable ? 12530 : 14190 ) : hasShadowCells
> ? 25320 : 12420, rs.getInt(3));
> assertEquals(oneCellPerColFamilyStorageScheme ? 13 : hasShadowCells ? 23
> : 12, rs.getInt(4));
> assertTrue(rs.next());
> assertEquals("B", rs.getString(1));
> assertEquals(oneCellPerColFamilyStorageScheme ? 25 : 20, rs.getInt(2));
> assertEquals(columnEncoded ? ( mutable ? 5600 : 7260 ) : hasShadowCells ?
> 11260 : 5540, rs.getInt(3));
> assertEquals(oneCellPerColFamilyStorageScheme ? 7 : hasShadowCells ? 10 :
> 5, rs.getInt(4));
> assertTrue(rs.next());
> assertEquals("C", rs.getString(1));
> assertEquals(25, rs.getInt(2));
> assertEquals(columnEncoded ? ( mutable ? 7005 : 7280 ) : hasShadowCells ?
> 14085 : 6930, rs.getInt(3));
> assertEquals(hasShadowCells ? 13 : 7, rs.getInt(4));
> assertTrue(rs.next());
> assertEquals("D", rs.getString(1));
> assertEquals(25, rs.getInt(2));
> assertEquals(columnEncoded ? ( mutable ? 7005 : 7280 ) : hasShadowCells ?
> 14085 : 6930, rs.getInt(3));
> assertEquals(hasShadowCells ? 13 : 7, rs.getInt(4));
> assertFalse(rs.next());
> // Disable stats
> conn.createStatement().execute("ALTER TABLE " + fullTableName +
> " SET " + PhoenixDatabaseMetaData.GUIDE_POSTS_WIDTH + "=0");
>
[jira] [Updated] (PHOENIX-3808) Implement chaos tests using HBase's hbase-it facility
[ https://issues.apache.org/jira/browse/PHOENIX-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-3808: Labels: hardening (was: ) > Implement chaos tests using HBase's hbase-it facility > - > > Key: PHOENIX-3808 > URL: https://issues.apache.org/jira/browse/PHOENIX-3808 > Project: Phoenix > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Archana Katiyar >Priority: Major > Labels: hardening > > Implement chaos tests using HBase's hbase-it facility. Especially, > correctness testing with an active server killing monkey policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5101) ScanningResultIterator getScanMetrics throws NPE
[
https://issues.apache.org/jira/browse/PHOENIX-5101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas D'Silva updated PHOENIX-5101:
Attachment: PHOENIX-5101.patch
> ScanningResultIterator getScanMetrics throws NPE
>
>
> Key: PHOENIX-5101
> URL: https://issues.apache.org/jira/browse/PHOENIX-5101
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.14.1
>Reporter: Reid Chan
>Assignee: Karan Mehta
>Priority: Blocker
> Fix For: 4.15.0, 5.1.0, 4.14.2
>
> Attachments: PHOENIX-5101.414-HBase-1.4.001.patch, PHOENIX-5101.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code}
> java.lang.NullPointerException
> at
> org.apache.phoenix.iterate.ScanningResultIterator.getScanMetrics(ScanningResultIterator.java:92)
> at
> org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:79)
> at
> org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:144)
> at
> org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42)
> at
> org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1439)
> at
> org.apache.phoenix.iterate.MergeSortResultIterator.close(MergeSortResultIterator.java:44)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.close(PhoenixResultSet.java:176)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:807)
> at
> org.apache.calcite.avatica.jdbc.JdbcResultSet.frame(JdbcResultSet.java:148)
> at
> org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:101)
> at
> org.apache.calcite.avatica.jdbc.JdbcResultSet.create(JdbcResultSet.java:81)
> at
> org.apache.calcite.avatica.jdbc.JdbcMeta.prepareAndExecute(JdbcMeta.java:759)
> at
> org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:206)
> at
> org.apache.calcite.avatica.remote.Service$PrepareAndExecuteRequest.accept(Service.java:927)
> at
> org.apache.calcite.avatica.remote.Service$PrepareAndExecuteRequest.accept(Service.java:879)
> at
> org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:94)
> at
> org.apache.calcite.avatica.remote.ProtobufHandler.apply(ProtobufHandler.java:46)
> at
> org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:123)
> at
> org.apache.calcite.avatica.server.AvaticaProtobufHandler$2.call(AvaticaProtobufHandler.java:121)
> at
> org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback$1.run(QueryServer.java:500)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> at
> org.apache.phoenix.queryserver.server.QueryServer$PhoenixDoAsCallback.doAsRemoteUser(QueryServer.java:497)
> at
> org.apache.calcite.avatica.server.HttpServer$Builder$1.doAsRemoteUser(HttpServer.java:884)
> at
> org.apache.calcite.avatica.server.AvaticaProtobufHandler.handle(AvaticaProtobufHandler.java:120)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:542)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (PHOENIX-5268) HBase 2.0.5 compatibility
Josh Elser created PHOENIX-5268:
---
Summary: HBase 2.0.5 compatibility
Key: PHOENIX-5268
URL: https://issues.apache.org/jira/browse/PHOENIX-5268
Project: Phoenix
Issue Type: Bug
Reporter: Josh Elser
Fix For: 5.1.0
HBASE-21754 introduces a new abstract method to RpcScheduler:
{{getMetaPriorityQueueLength()}}
This means that Phoenix does not build against HBase 2.0.5.
FYI [~twdsi...@gmail.com], [~Jaanai].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release of Apache Phoenix 4.14.2 RC0
-1 PHOENIX-5101 and PHOENIX-5266 are blockers. On Tue, Apr 23, 2019 at 5:24 PM William Shen wrote: > The exception seems to be unique to our use case where the server uses > swagger and has a conflict on guava with the version of guava needed in the > HBase client. I was able to get the code running by removing swagger. > > However, using 4.12.2-HBase-1.2, I am still running into the NPE in > https://issues.apache.org/jira/browse/PHOENIX-5101 > I had left a comment > < > https://issues.apache.org/jira/browse/PHOENIX-5101?focusedCommentId=16812747=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16812747 > > > on the ticket previously because the fix might not work for HBase 1.2 and > 1.3 > > Stacktrace: > > Caused by: java.lang.NullPointerException: null >at > org.apache.phoenix.iterate.ScanningResultIterator.getScanMetrics(ScanningResultIterator.java:100) > ~[phoenix-core-4.14.2-HBase-1.2.jar:4.14.2-HBase-1.2] >at > org.apache.phoenix.iterate.ScanningResultIterator.close(ScanningResultIterator.java:80) > ~[phoenix-core-4.14.2-HBase-1.2.jar:4.14.2-HBase-1.2] >at > org.apache.phoenix.iterate.TableResultIterator.close(TableResultIterator.java:144) > ~[phoenix-core-4.14.2-HBase-1.2.jar:4.14.2-HBase-1.2] >at > org.apache.phoenix.iterate.LookAheadResultIterator$1.close(LookAheadResultIterator.java:42) > ~[phoenix-core-4.14.2-HBase-1.2.jar:4.14.2-HBase-1.2] >at > org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1442) > ~[phoenix-core-4.14.2-HBase-1.2.jar:4.14.2-HBase-1.2] >at > org.apache.phoenix.iterate.RoundRobinResultIterator.close(RoundRobinResultIterator.java:125) > ~[phoenix-core-4.14.2-HBase-1.2.jar:4.14.2-HBase-1.2] >... 64 common frames omitted > > > On Mon, Apr 22, 2019 at 5:11 PM William Shen > wrote: > > > Seems to be due to https://issues.apache.org/jira/browse/HBASE-14963 > which > > is not in HBase-1.2 > > > > On Mon, Apr 22, 2019 at 4:38 PM William Shen > > > wrote: > > > >> the bottom of the stacktrace: > >> Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: > >> java.lang.IllegalAccessError: tried to access method > >> com.google.common.base.Stopwatch.()V from class > >> org.apache.hadoop.hbase.zookeeper.MetaTableLocator > >> at > >> > org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239) > >> at > >> > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212) > >> at > >> > org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327) > >> at > >> > org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302) > >> at > >> > org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167) > >> at > >> > org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:162) > >> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797) > >> at > >> > org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602) > >> at > >> > org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366) > >> at > >> > org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406) > >> at > >> > org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1097) > >> ... 30 more > >> Caused by: java.lang.IllegalAccessError: tried to access method > >> com.google.common.base.Stopwatch.()V from class > >> org.apache.hadoop.hbase.zookeeper.MetaTableLocator > >> at > >> > org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:604) > >> at > >> > org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588) > >> at > >> > org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561) > >> at > >> > org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61) > >> at > >> > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211) > >> at > >> > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178) > >> at > >> > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305) > >> at > >> > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156) > >> at > >> > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60) > >> at > >> > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210) > >> ... 39 more > >> > >> On Mon, Apr 22, 2019 at 4:26 PM William Shen < > wills...@marinsoftware.com> > >> wrote: > >> > >>> Not sure if it's a local issue unique to my setup, but when I set up my > >>> Java client to use 4.14.2, I encounter the
[jira] [Updated] (PHOENIX-5266) Client can only write on Index Table and skip data table if failure happens because of region split/move etc
[ https://issues.apache.org/jira/browse/PHOENIX-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas D'Silva updated PHOENIX-5266: Fix Version/s: 4.14.2 > Client can only write on Index Table and skip data table if failure happens > because of region split/move etc > > > Key: PHOENIX-5266 > URL: https://issues.apache.org/jira/browse/PHOENIX-5266 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.1, 4.14.2 >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Blocker > Fix For: 4.14.2 > > > With Phoenix 4.14.1 client, There is a scenario where client would skip data > table write but do successful index table write. In this case, we should > treat it as Data loss scenario. > > Relevant code path :- > [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L994-L1043] > [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L1089-L1109] > > Here is what happens :- > * Consider below assumptions for scenario :- > ** max no row in single batch = 100 > ** max size of batch = 2 MB > * When client faces SQLException Code 1121, it sets variable > shouldRetryIndexedMutation=true. > * In scenarios where client sends batch of 100 rows only as per > configuration, but batch size is >2 MB, MutationState.java#991 will split > this 100 row batch into multiple smaller batches which are <2MB. > ** MutationState.java#991 :- > [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L991] > * Suppose there are 5 batches of 20 rows but client faces 1121 > SQLExceptionCode on 2nd batch , then it will set > shouldRetryIndexedMutation=true and it will retry all 5 batches again with > only Index updates. This will results in rows missing from Data table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5258) Add support to parse header from the input CSV file as input columns for CsvBulkLoadTool
[ https://issues.apache.org/jira/browse/PHOENIX-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Vithani updated PHOENIX-5258: -- Attachment: PHOENIX-5258-4.x-HBase-1.4.patch > Add support to parse header from the input CSV file as input columns for > CsvBulkLoadTool > > > Key: PHOENIX-5258 > URL: https://issues.apache.org/jira/browse/PHOENIX-5258 > Project: Phoenix > Issue Type: Improvement >Reporter: Prashant Vithani >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-5258-4.x-HBase-1.4.patch, > PHOENIX-5258-master.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, CsvBulkLoadTool does not support reading header from the input csv > and expects the content of the csv to match with the table schema. The > support for the header can be added to dynamically map the schema with the > header. > The proposed solution is to introduce another option for the tool `–header`. > If this option is passed, the input columns list is constructed by reading > the first line of the input CSV file. > * If there is only one file, read the header from the first line and > generate the `ColumnInfo` list. > * If there are multiple files, read the header from all the files, and throw > an error if the headers across files do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5258) Add support to parse header from the input CSV file as input columns for CsvBulkLoadTool
[ https://issues.apache.org/jira/browse/PHOENIX-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Vithani updated PHOENIX-5258: -- Fix Version/s: (was: 4.14.2) 4.15.0 > Add support to parse header from the input CSV file as input columns for > CsvBulkLoadTool > > > Key: PHOENIX-5258 > URL: https://issues.apache.org/jira/browse/PHOENIX-5258 > Project: Phoenix > Issue Type: Improvement >Reporter: Prashant Vithani >Priority: Minor > Fix For: 4.15.0, 5.1.0 > > Attachments: PHOENIX-5258-master.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, CsvBulkLoadTool does not support reading header from the input csv > and expects the content of the csv to match with the table schema. The > support for the header can be added to dynamically map the schema with the > header. > The proposed solution is to introduce another option for the tool `–header`. > If this option is passed, the input columns list is constructed by reading > the first line of the input CSV file. > * If there is only one file, read the header from the first line and > generate the `ColumnInfo` list. > * If there are multiple files, read the header from all the files, and throw > an error if the headers across files do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5250) The accumulated wal files can't be cleaned
[ https://issues.apache.org/jira/browse/PHOENIX-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated PHOENIX-5250: Description: Because of the modification of HBASE-20781, the faked WALEdits won't be filtered, all WALEdits will be saved into Memstore with a status that inMemstore is true(uses WAL->append method). !image-2019-04-19-21-31-27-888.png|width=755,height=310! The family array of IndexedKeyValue is WALEdit.METAFAMILY that is used to describe a fake WALEdit, and it will put into Memstore with WALedits of data table during sync global index. WAL files can't be cleaned, except for restarting RS, Many WAL files will decrease the percent of disk free. !https://gw.alicdn.com/tfscom/TB1n3cDQVzqK1RjSZFCXXbbxVXa.png|width=422,height=159! was: Because of the modification of HBASE-20781, the faked WALEdits won't be filtered, all WALEdits will be saved into Memstore with a status that inMemstore is true(uses WAL->append method). !image-2019-04-19-21-31-27-888.png|width=755,height=310! The family array of IndexedKeyValue is WALEdit.METAFAMILY that is used to describe a fake WALEdit, and it will put into Memstore with WALedits of data table during sync global index. WAL files can't be cleaned, expect for resarting RS, Many WAL files will decrease the percent of disk free. !https://gw.alicdn.com/tfscom/TB1n3cDQVzqK1RjSZFCXXbbxVXa.png|width=422,height=159! > The accumulated wal files can't be cleaned > -- > > Key: PHOENIX-5250 > URL: https://issues.apache.org/jira/browse/PHOENIX-5250 > Project: Phoenix > Issue Type: Bug >Reporter: jaanai >Priority: Blocker > Fix For: 5.1.0 > > Attachments: image-2019-04-19-21-31-27-888.png > > > Because of the modification of HBASE-20781, the faked WALEdits won't be > filtered, all WALEdits will be saved into Memstore with a status that > inMemstore is true(uses WAL->append method). > !image-2019-04-19-21-31-27-888.png|width=755,height=310! > The family array of IndexedKeyValue is WALEdit.METAFAMILY that is used to > describe a fake WALEdit, and it will put into Memstore with WALedits of data > table during sync global index. > WAL files can't be cleaned, except for restarting RS, Many WAL files will > decrease the percent of disk free. > !https://gw.alicdn.com/tfscom/TB1n3cDQVzqK1RjSZFCXXbbxVXa.png|width=422,height=159! > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5267) With namespaces enabled Phoenix client times out with high loads
[
https://issues.apache.org/jira/browse/PHOENIX-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kiran Kumar Maturi updated PHOENIX-5267:
Description:
Steps to reproduce:
* Enable namespaces for phoenix 4.14.1 and hbase 1.3
* Run high load using pherf client with 48 threads
After sometime the client hangs. and gives timeout exception
{code:java}
[pool-1-thread-1] WARN org.apache.phoenix.pherf.workload.WriteWorkload -
java.util.concurrent.ExecutionException:
org.apache.phoenix.exception.PhoenixIOException: callTimeout=120,
callDuration=1238263: Call to failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857, waitTime=120001,
operationTimeout=12 expired. row '^@TEST^@TABLE' on table 'SYSTEM:CATALOG'
at region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254.,
hostname=, seqNum=2
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.phoenix.pherf.workload.WriteWorkload.waitForBatches(WriteWorkload.java:239)
at
org.apache.phoenix.pherf.workload.WriteWorkload.exec(WriteWorkload.java:189)
at
org.apache.phoenix.pherf.workload.WriteWorkload.access$100(WriteWorkload.java:56)
at
org.apache.phoenix.pherf.workload.WriteWorkload$1.run(WriteWorkload.java:165)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.phoenix.exception.PhoenixIOException:
callTimeout=120, callDuration=1238263: Call to failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857,
waitTime=120001, operationTimeout=12 expired. row '^@TEST^@TABLE' on table
'SYSTEM:CATALOG' at
region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254.,
hostname=, seqNum=2
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1379)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1343)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1560)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:644)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:538)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:530)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:526)
at
org.apache.phoenix.execute.MutationState.validateAndGetServerTimestamp(MutationState.java:755)
at
org.apache.phoenix.execute.MutationState.validateAll(MutationState.java:743)
at org.apache.phoenix.execute.MutationState.send(MutationState.java:875)
at org.apache.phoenix.execute.MutationState.send(MutationState.java:1360)
at org.apache.phoenix.execute.MutationState.commit(MutationState.java:1183)
at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:670)
at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:666)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:666)
at
org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:297)
at
org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:256)
{code}
was:
Steps to reproduce:
* Enable namespaces for phoenix 4.14.1 and hbase 1.3
* Run high load using pherf client with 48 threads
After sometime the client hangs. and gives timeout exception
{code:java}
[pool-1-thread-1] WARN org.apache.phoenix.pherf.workload.WriteWorkload -
java.util.concurrent.ExecutionException:
org.apache.phoenix.exception.PhoenixIOException: callTimeout=120,
callDuration=1238263: Call to failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857, waitTime=120001,
operationTimeout=12 expired. row '^@test^@table' on table 'SYSTEM:CATALOG'
at region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254.,
hostname=, seqNum=2
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.phoenix.pherf.workload.WriteWorkload.waitForBatches(WriteWorkload.java:239)
at
org.apache.phoenix.pherf.workload.WriteWorkload.exec(WriteWorkload.java:189)
at
[jira] [Created] (PHOENIX-5267) With namespaces enabled Phoenix client times out with high loads
Kiran Kumar Maturi created PHOENIX-5267:
---
Summary: With namespaces enabled Phoenix client times out with
high loads
Key: PHOENIX-5267
URL: https://issues.apache.org/jira/browse/PHOENIX-5267
Project: Phoenix
Issue Type: Bug
Affects Versions: 4.14.1
Reporter: Kiran Kumar Maturi
Steps to reproduce:
* Enable namespaces for phoenix 4.14.1 and hbase 1.3
* Run high load using pherf client with 48 threads
After sometime the client hangs. and gives timeout exception
{code:java}
[pool-1-thread-1] WARN org.apache.phoenix.pherf.workload.WriteWorkload -
java.util.concurrent.ExecutionException:
org.apache.phoenix.exception.PhoenixIOException: callTimeout=120,
callDuration=1238263: Call to failed on local exception:
org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857, waitTime=120001,
operationTimeout=12 expired. row '^@test^@table' on table 'SYSTEM:CATALOG'
at region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254.,
hostname=, seqNum=2
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.phoenix.pherf.workload.WriteWorkload.waitForBatches(WriteWorkload.java:239)
at
org.apache.phoenix.pherf.workload.WriteWorkload.exec(WriteWorkload.java:189)
at
org.apache.phoenix.pherf.workload.WriteWorkload.access$100(WriteWorkload.java:56)
at
org.apache.phoenix.pherf.workload.WriteWorkload$1.run(WriteWorkload.java:165)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.phoenix.exception.PhoenixIOException:
callTimeout=120, callDuration=1238263: Call to failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=857,
waitTime=120001, operationTimeout=12 expired. row '^@TEST^@TABLE' on table
'SYSTEM:CATALOG' at
region=SYSTEM:CATALOG,1556024429507.0f80d6de0a002d1421b8fd384e956254.,
hostname=, seqNum=2
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:144)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1379)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1343)
at
org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1560)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:644)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:538)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:530)
at
org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:526)
at
org.apache.phoenix.execute.MutationState.validateAndGetServerTimestamp(MutationState.java:755)
at
org.apache.phoenix.execute.MutationState.validateAll(MutationState.java:743)
at org.apache.phoenix.execute.MutationState.send(MutationState.java:875)
at org.apache.phoenix.execute.MutationState.send(MutationState.java:1360)
at org.apache.phoenix.execute.MutationState.commit(MutationState.java:1183)
at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:670)
at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:666)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:666)
at
org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:297)
at
org.apache.phoenix.pherf.workload.WriteWorkload$2.call(WriteWorkload.java:256)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (PHOENIX-5258) Add support to parse header from the input CSV file as input columns for CsvBulkLoadTool
[ https://issues.apache.org/jira/browse/PHOENIX-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Vithani updated PHOENIX-5258: -- Attachment: (was: PHOENIX-5258-master.patch) > Add support to parse header from the input CSV file as input columns for > CsvBulkLoadTool > > > Key: PHOENIX-5258 > URL: https://issues.apache.org/jira/browse/PHOENIX-5258 > Project: Phoenix > Issue Type: Improvement >Reporter: Prashant Vithani >Priority: Minor > Fix For: 5.1.0, 4.14.2 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, CsvBulkLoadTool does not support reading header from the input csv > and expects the content of the csv to match with the table schema. The > support for the header can be added to dynamically map the schema with the > header. > The proposed solution is to introduce another option for the tool `–header`. > If this option is passed, the input columns list is constructed by reading > the first line of the input CSV file. > * If there is only one file, read the header from the first line and > generate the `ColumnInfo` list. > * If there are multiple files, read the header from all the files, and throw > an error if the headers across files do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (PHOENIX-5258) Add support to parse header from the input CSV file as input columns for CsvBulkLoadTool
[ https://issues.apache.org/jira/browse/PHOENIX-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Vithani updated PHOENIX-5258: -- Attachment: PHOENIX-5258-master.patch > Add support to parse header from the input CSV file as input columns for > CsvBulkLoadTool > > > Key: PHOENIX-5258 > URL: https://issues.apache.org/jira/browse/PHOENIX-5258 > Project: Phoenix > Issue Type: Improvement >Reporter: Prashant Vithani >Priority: Minor > Fix For: 5.1.0, 4.14.2 > > Attachments: PHOENIX-5258-master.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, CsvBulkLoadTool does not support reading header from the input csv > and expects the content of the csv to match with the table schema. The > support for the header can be added to dynamically map the schema with the > header. > The proposed solution is to introduce another option for the tool `–header`. > If this option is passed, the input columns list is constructed by reading > the first line of the input CSV file. > * If there is only one file, read the header from the first line and > generate the `ColumnInfo` list. > * If there are multiple files, read the header from all the files, and throw > an error if the headers across files do not match. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [ANNOUNCE] New Phoenix committer: Mihir Monani
Thanks you everyone. :) On Wed, May 1, 2019 at 9:13 AM Geoffrey Jacoby wrote: > Congratulations, Mihir! > > On Tue, Apr 30, 2019 at 6:01 PM Ankit Singhal > wrote: > > > Congratulations and Welcome Mihir !! > > > > On Tue, Apr 30, 2019 at 5:49 PM Andrew Purtell > > wrote: > > > > > Congratulations Mihir. > > > > > > On Sat, Apr 27, 2019 at 5:42 PM Thomas D'Silva > > wrote: > > > > > > > On behalf of the Apache Phoenix PMC, I am pleased to announce that > > > > Mihir Monani has accepted our invitation to become a committer. > > > > Mihir has done some nice work fixing several bugs related to > > indexing[1]. > > > > > > > > Please welcome him to the Apache Phoenix team. > > > > > > > > Thanks, > > > > Thomas > > > > > > > > [1] > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/PHOENIX-5199?jql=project%20%3D%20PHOENIX%20AND%20assignee%3D%22mihir6692%22%20AND%20status%3DResolved > > > > > > > > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > > decrepit hands > > >- A23, Crosstalk > > > > > > -- Mihir Monani (+91)-9429473434
[jira] [Updated] (PHOENIX-5266) Client can only write on Index Table and skip data table if failure happens because of region split/move etc
[ https://issues.apache.org/jira/browse/PHOENIX-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihir Monani updated PHOENIX-5266: -- Description: With Phoenix 4.14.1 client, There is a scenario where client would skip data table write but do successful index table write. In this case, we should treat it as Data loss scenario. Relevant code path :- [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L994-L1043] [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L1089-L1109] Here is what happens :- * Consider below assumptions for scenario :- ** max no row in single batch = 100 ** max size of batch = 2 MB * When client faces SQLException Code 1121, it sets variable shouldRetryIndexedMutation=true. * In scenarios where client sends batch of 100 rows only as per configuration, but batch size is >2 MB, MutationState.java#991 will split this 100 row batch into multiple smaller batches which are <2MB. ** MutationState.java#991 :- [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L991] * Suppose there are 5 batches of 20 rows but client faces 1121 SQLExceptionCode on 2nd batch , then it will set shouldRetryIndexedMutation=true and it will retry all 5 batches again with only Index updates. This will results in rows missing from Data table. was: With Phoenix 4.14.1 client, There is a scenario where client would skip data table write but do successful index table write. In this case, we should treat it as Data loss scenario. Relevant code path :- https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L994-L1043 https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L1089-L1109 Here is what happens :- * Consider below assumptions for scenario :- ** max no row in single batch = 100 ** max size of batch = 2 MB * When client faces SQLException Code 1121, it sets variable shouldRetryIndexedMutation=true. * In scenarios where client sends batch of 100 rows only as per configuration, but batch size is >2 MB, MutationState.java#978 will split this 100 row batch into multiple smaller batches which are <2MB. * Suppose there are 5 batches of 20 rows but client faces 1121 SQLExceptionCode on 2nd batch , then it will set shouldRetryIndexedMutation=true and it will retry all 5 batches again with only Index updates. This will results in rows missing from Data table. > Client can only write on Index Table and skip data table if failure happens > because of region split/move etc > > > Key: PHOENIX-5266 > URL: https://issues.apache.org/jira/browse/PHOENIX-5266 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.14.1, 4.14.2 >Reporter: Mihir Monani >Assignee: Mihir Monani >Priority: Blocker > > With Phoenix 4.14.1 client, There is a scenario where client would skip data > table write but do successful index table write. In this case, we should > treat it as Data loss scenario. > > Relevant code path :- > [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L994-L1043] > [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L1089-L1109] > > Here is what happens :- > * Consider below assumptions for scenario :- > ** max no row in single batch = 100 > ** max size of batch = 2 MB > * When client faces SQLException Code 1121, it sets variable > shouldRetryIndexedMutation=true. > * In scenarios where client sends batch of 100 rows only as per > configuration, but batch size is >2 MB, MutationState.java#991 will split > this 100 row batch into multiple smaller batches which are <2MB. > ** MutationState.java#991 :- > [https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L991] > * Suppose there are 5 batches of 20 rows but client faces 1121 > SQLExceptionCode on 2nd batch , then it will set > shouldRetryIndexedMutation=true and it will retry all 5 batches again with > only Index updates. This will results in rows missing from Data table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (PHOENIX-5266) Client can only write on Index Table and skip data table if failure happens because of region split/move etc
Mihir Monani created PHOENIX-5266: - Summary: Client can only write on Index Table and skip data table if failure happens because of region split/move etc Key: PHOENIX-5266 URL: https://issues.apache.org/jira/browse/PHOENIX-5266 Project: Phoenix Issue Type: Bug Affects Versions: 4.14.1, 4.14.2 Reporter: Mihir Monani Assignee: Mihir Monani With Phoenix 4.14.1 client, There is a scenario where client would skip data table write but do successful index table write. In this case, we should treat it as Data loss scenario. Relevant code path :- https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L994-L1043 https://github.com/apache/phoenix/blob/4.x-HBase-1.3/phoenix-core/src/main/java/org/apache/phoenix/execute/MutationState.java#L1089-L1109 Here is what happens :- * Consider below assumptions for scenario :- ** max no row in single batch = 100 ** max size of batch = 2 MB * When client faces SQLException Code 1121, it sets variable shouldRetryIndexedMutation=true. * In scenarios where client sends batch of 100 rows only as per configuration, but batch size is >2 MB, MutationState.java#978 will split this 100 row batch into multiple smaller batches which are <2MB. * Suppose there are 5 batches of 20 rows but client faces 1121 SQLExceptionCode on 2nd batch , then it will set shouldRetryIndexedMutation=true and it will retry all 5 batches again with only Index updates. This will results in rows missing from Data table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
