[jira] [Commented] (PHOENIX-6640) Bump Avatica to 1.20 in PQS
[ https://issues.apache.org/jira/browse/PHOENIX-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486230#comment-17486230 ] Istvan Toth commented on PHOENIX-6640: -- At first try tests are failing / hanging. > Bump Avatica to 1.20 in PQS > --- > > Key: PHOENIX-6640 > URL: https://issues.apache.org/jira/browse/PHOENIX-6640 > Project: Phoenix > Issue Type: Improvement > Components: queryserver >Affects Versions: queryserver-6.0.1 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6639) Read repair of a table after cutover (transform is complete and table is switched)
[ https://issues.apache.org/jira/browse/PHOENIX-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486178#comment-17486178 ] ASF GitHub Bot commented on PHOENIX-6639: - gokceni commented on pull request #1391: URL: https://github.com/apache/phoenix/pull/1391#issuecomment-1028502607 @gjacoby126 @tkhurana -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Read repair of a table after cutover (transform is complete and table is > switched) > -- > > Key: PHOENIX-6639 > URL: https://issues.apache.org/jira/browse/PHOENIX-6639 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on pull request #1391: PHOENIX-6639 Read repair for transforming table
gokceni commented on pull request #1391: URL: https://github.com/apache/phoenix/pull/1391#issuecomment-1028502607 @gjacoby126 @tkhurana -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6622) TransformMonitor should orchestrate transform and do retries
[ https://issues.apache.org/jira/browse/PHOENIX-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486175#comment-17486175 ] ASF GitHub Bot commented on PHOENIX-6622: - gokceni closed pull request #1373: URL: https://github.com/apache/phoenix/pull/1373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TransformMonitor should orchestrate transform and do retries > > > Key: PHOENIX-6622 > URL: https://issues.apache.org/jira/browse/PHOENIX-6622 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6622) TransformMonitor should orchestrate transform and do retries
[ https://issues.apache.org/jira/browse/PHOENIX-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486176#comment-17486176 ] ASF GitHub Bot commented on PHOENIX-6622: - gokceni commented on pull request #1373: URL: https://github.com/apache/phoenix/pull/1373#issuecomment-1028494706 Merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > TransformMonitor should orchestrate transform and do retries > > > Key: PHOENIX-6622 > URL: https://issues.apache.org/jira/browse/PHOENIX-6622 > Project: Phoenix > Issue Type: Sub-task >Reporter: Gokcen Iskender >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] gokceni commented on pull request #1373: PHOENIX-6622 Transform monitor
gokceni commented on pull request #1373: URL: https://github.com/apache/phoenix/pull/1373#issuecomment-1028494706 Merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [phoenix] gokceni closed pull request #1373: PHOENIX-6622 Transform monitor
gokceni closed pull request #1373: URL: https://github.com/apache/phoenix/pull/1373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-628) Support native JSON data type
[ https://issues.apache.org/jira/browse/PHOENIX-628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486078#comment-17486078 ] Istvan Toth commented on PHOENIX-628: - Sorry, [~giskender] , I missed that. > Support native JSON data type > - > > Key: PHOENIX-628 > URL: https://issues.apache.org/jira/browse/PHOENIX-628 > Project: Phoenix > Issue Type: Task >Affects Versions: 4.4.0 >Reporter: James R. Taylor > Labels: JSON, Java, SQL > Fix For: 4.4.1 > > Attachments: JSON Support for Phoenix.docx, Screen Shot 2022-02-02 at > 12.23.24 PM.png > > > MongoDB and PostGres do some interesting things with JSON. We should look at > adding similar support. For a detailed description, see JSONB support in > Postgres: > http://www.craigkerstiens.com/2014/03/24/Postgres-9.4-Looking-up > http://www.depesz.com/2014/03/25/waiting-for-9-4-introduce-jsonb-a-structured-format-for-storing-json/ > http://michael.otacoo.com/postgresql-2/manipulating-jsonb-data-with-key-unique/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-628) Support native JSON data type
[ https://issues.apache.org/jira/browse/PHOENIX-628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486074#comment-17486074 ] Gokcen Iskender commented on PHOENIX-628: - [~stoty] I am not able to share globally but it is already attached to the jira. !Screen Shot 2022-02-02 at 12.23.24 PM.png! > Support native JSON data type > - > > Key: PHOENIX-628 > URL: https://issues.apache.org/jira/browse/PHOENIX-628 > Project: Phoenix > Issue Type: Task >Affects Versions: 4.4.0 >Reporter: James R. Taylor > Labels: JSON, Java, SQL > Fix For: 4.4.1 > > Attachments: JSON Support for Phoenix.docx, Screen Shot 2022-02-02 at > 12.23.24 PM.png > > > MongoDB and PostGres do some interesting things with JSON. We should look at > adding similar support. For a detailed description, see JSONB support in > Postgres: > http://www.craigkerstiens.com/2014/03/24/Postgres-9.4-Looking-up > http://www.depesz.com/2014/03/25/waiting-for-9-4-introduce-jsonb-a-structured-format-for-storing-json/ > http://michael.otacoo.com/postgresql-2/manipulating-jsonb-data-with-key-unique/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (PHOENIX-6441) Remove TSOMockModule reference from OmidTransactionProvider
[ https://issues.apache.org/jira/browse/PHOENIX-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486062#comment-17486062 ] ASF GitHub Bot commented on PHOENIX-6441: - stoty closed pull request #1390: URL: https://github.com/apache/phoenix/pull/1390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove TSOMockModule reference from OmidTransactionProvider > --- > > Key: PHOENIX-6441 > URL: https://issues.apache.org/jira/browse/PHOENIX-6441 > Project: Phoenix > Issue Type: Bug > Components: core, omid >Affects Versions: 5.2.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 4.17.0, 5.2.0, 4.16.2, 5.1.3 > > > OmidTransactionProvider references TSOMockModule which is a test dependency. > It seems to be used only by tests. > Non-test code should not depend on test artifacts. > Move the functionality into test code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] stoty closed pull request #1390: PHOENIX-6441 Remove TSOMockModule reference from OmidTransactionProvi…
stoty closed pull request #1390: URL: https://github.com/apache/phoenix/pull/1390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[
https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485876#comment-17485876
]
ASF GitHub Bot commented on PHOENIX-6632:
-
stoty commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797705703
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixScan.java
##
@@ -29,134 +29,104 @@
import org.apache.phoenix.mapreduce.PhoenixInputSplit;
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.spark.FilterExpressionCompiler;
-import org.apache.phoenix.spark.SparkSchemaUtil;
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource;
import org.apache.phoenix.util.ColumnInfo;
import org.apache.phoenix.util.PhoenixRuntime;
import org.apache.phoenix.util.QueryUtil;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.Filter;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.reader.InputPartition;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.Scan;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.PartitionReaderFactory;
import org.apache.spark.sql.types.StructType;
-import scala.Tuple3;
-import scala.collection.JavaConverters;
-import scala.collection.Seq;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import java.sql.Connection;
import java.sql.DriverManager;
-import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import java.util.Optional;
import java.util.Properties;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
-public class PhoenixDataSourceReader implements DataSourceReader,
SupportsPushDownFilters,
-SupportsPushDownRequiredColumns {
+public class PhoenixScan implements Scan, Batch {
-private final DataSourceOptions options;
-private final String tableName;
+private final StructType schema;
+private final CaseInsensitiveStringMap options;
private final String zkUrl;
-private final boolean dateAsTimestamp;
private final Properties overriddenProps;
+private PhoenixDataSourceReadOptions phoenixDataSourceOptions;
+private final String tableName;
+private String currentScnValue;
+private String tenantId;
+private boolean splitByStats;
+private final String whereClause;
-private StructType schema;
-private Filter[] pushedFilters = new Filter[]{};
-// derived from pushedFilters
-private String whereClause;
-
-public PhoenixDataSourceReader(DataSourceOptions options) {
-if (!options.tableName().isPresent()) {
-throw new RuntimeException("No Phoenix option " +
DataSourceOptions.TABLE_KEY + " defined");
-}
-if (!options.get(PhoenixDataSource.ZOOKEEPER_URL).isPresent()) {
-throw new RuntimeException("No Phoenix option " +
PhoenixDataSource.ZOOKEEPER_URL + " defined");
-}
+PhoenixScan(StructType schema, CaseInsensitiveStringMap options, String
whereClause) {
+this.schema = schema;
this.options = options;
-this.tableName = options.tableName().get();
-this.zkUrl = options.get(PhoenixDataSource.ZOOKEEPER_URL).get();
-this.dateAsTimestamp = options.getBoolean("dateAsTimestamp", false);
+this.whereClause = whereClause;
this.overriddenProps = extractPhoenixHBaseConfFromOptions(options);
-setSchema();
+this.zkUrl = options.get(PhoenixDataSource.ZOOKEEPER_URL);
+tableName = options.get("table");
}
-/**
- * Sets the schema using all the table columns before any column pruning
has been done
- */
-private void setSchema() {
-try (Connection conn = DriverManager.getConnection(
-JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl,
overriddenProps)) {
-List columnInfos =
PhoenixRuntime.generateColumnInfo(conn, tableName, null);
-Seq columnInfoSeq =
JavaConverters.asScalaIteratorConverter(columnInfos.iterator()).asScala().toSeq();
-schema =
SparkSchemaUtil.phoenixSchemaToCatalystSchema(columnInfoSeq, dateAsTimestamp);
+private void
[GitHub] [phoenix-connectors] stoty commented on a change in pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
stoty commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797705703
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixScan.java
##
@@ -29,134 +29,104 @@
import org.apache.phoenix.mapreduce.PhoenixInputSplit;
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.spark.FilterExpressionCompiler;
-import org.apache.phoenix.spark.SparkSchemaUtil;
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource;
import org.apache.phoenix.util.ColumnInfo;
import org.apache.phoenix.util.PhoenixRuntime;
import org.apache.phoenix.util.QueryUtil;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.Filter;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.reader.InputPartition;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.Scan;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.PartitionReaderFactory;
import org.apache.spark.sql.types.StructType;
-import scala.Tuple3;
-import scala.collection.JavaConverters;
-import scala.collection.Seq;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import java.sql.Connection;
import java.sql.DriverManager;
-import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import java.util.Optional;
import java.util.Properties;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
-public class PhoenixDataSourceReader implements DataSourceReader,
SupportsPushDownFilters,
-SupportsPushDownRequiredColumns {
+public class PhoenixScan implements Scan, Batch {
-private final DataSourceOptions options;
-private final String tableName;
+private final StructType schema;
+private final CaseInsensitiveStringMap options;
private final String zkUrl;
-private final boolean dateAsTimestamp;
private final Properties overriddenProps;
+private PhoenixDataSourceReadOptions phoenixDataSourceOptions;
+private final String tableName;
+private String currentScnValue;
+private String tenantId;
+private boolean splitByStats;
+private final String whereClause;
-private StructType schema;
-private Filter[] pushedFilters = new Filter[]{};
-// derived from pushedFilters
-private String whereClause;
-
-public PhoenixDataSourceReader(DataSourceOptions options) {
-if (!options.tableName().isPresent()) {
-throw new RuntimeException("No Phoenix option " +
DataSourceOptions.TABLE_KEY + " defined");
-}
-if (!options.get(PhoenixDataSource.ZOOKEEPER_URL).isPresent()) {
-throw new RuntimeException("No Phoenix option " +
PhoenixDataSource.ZOOKEEPER_URL + " defined");
-}
+PhoenixScan(StructType schema, CaseInsensitiveStringMap options, String
whereClause) {
+this.schema = schema;
this.options = options;
-this.tableName = options.tableName().get();
-this.zkUrl = options.get(PhoenixDataSource.ZOOKEEPER_URL).get();
-this.dateAsTimestamp = options.getBoolean("dateAsTimestamp", false);
+this.whereClause = whereClause;
this.overriddenProps = extractPhoenixHBaseConfFromOptions(options);
-setSchema();
+this.zkUrl = options.get(PhoenixDataSource.ZOOKEEPER_URL);
+tableName = options.get("table");
}
-/**
- * Sets the schema using all the table columns before any column pruning
has been done
- */
-private void setSchema() {
-try (Connection conn = DriverManager.getConnection(
-JDBC_PROTOCOL + JDBC_PROTOCOL_SEPARATOR + zkUrl,
overriddenProps)) {
-List columnInfos =
PhoenixRuntime.generateColumnInfo(conn, tableName, null);
-Seq columnInfoSeq =
JavaConverters.asScalaIteratorConverter(columnInfos.iterator()).asScala().toSeq();
-schema =
SparkSchemaUtil.phoenixSchemaToCatalystSchema(columnInfoSeq, dateAsTimestamp);
+private void populateOverriddenProperties(){
+currentScnValue =
options.get(PhoenixConfigurationUtil.CURRENT_SCN_VALUE);
+tenantId = options.get(PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID);
+// Generate splits based off statistics, or just region
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[
https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485874#comment-17485874
]
ASF GitHub Bot commented on PHOENIX-6632:
-
stoty commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797649170
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/PhoenixDataSource.java
##
@@ -17,42 +17,76 @@
*/
package org.apache.phoenix.spark.datasource.v2;
-import java.util.Optional;
-import java.util.Properties;
-
-import org.apache.phoenix.spark.datasource.v2.reader.PhoenixDataSourceReader;
-import org.apache.phoenix.spark.datasource.v2.writer.PhoenixDataSourceWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.spark.sql.SaveMode;
+import org.apache.phoenix.spark.SparkSchemaUtil;
+import org.apache.phoenix.util.ColumnInfo;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.spark.sql.connector.catalog.Table;
+import org.apache.spark.sql.connector.catalog.TableProvider;
+import org.apache.spark.sql.connector.expressions.Transform;
import org.apache.spark.sql.sources.DataSourceRegister;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.DataSourceV2;
-import org.apache.spark.sql.sources.v2.ReadSupport;
-import org.apache.spark.sql.sources.v2.WriteSupport;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
+import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
/**
* Implements the DataSourceV2 api to read and write from Phoenix tables
*/
-public class PhoenixDataSource implements DataSourceV2, ReadSupport,
WriteSupport, DataSourceRegister {
+public class PhoenixDataSource implements TableProvider, DataSourceRegister {
private static final Logger logger =
LoggerFactory.getLogger(PhoenixDataSource.class);
+public static final String TABLE_KEY = "table";
public static final String SKIP_NORMALIZING_IDENTIFIER =
"skipNormalizingIdentifier";
public static final String ZOOKEEPER_URL = "zkUrl";
public static final String PHOENIX_CONFIGS = "phoenixconfigs";
+protected StructType schema;
+private CaseInsensitiveStringMap options;
@Override
-public DataSourceReader createReader(DataSourceOptions options) {
-return new PhoenixDataSourceReader(options);
+public StructType inferSchema(CaseInsensitiveStringMap options){
+if (options.get("table") == null) {
Review comment:
use the new TABLE_KEY constant everywhere
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixScan.java
##
@@ -29,134 +29,104 @@
import org.apache.phoenix.mapreduce.PhoenixInputSplit;
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.spark.FilterExpressionCompiler;
-import org.apache.phoenix.spark.SparkSchemaUtil;
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource;
import org.apache.phoenix.util.ColumnInfo;
import org.apache.phoenix.util.PhoenixRuntime;
import org.apache.phoenix.util.QueryUtil;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.Filter;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.reader.InputPartition;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.Scan;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.PartitionReaderFactory;
import org.apache.spark.sql.types.StructType;
-import scala.Tuple3;
-import scala.collection.JavaConverters;
-import scala.collection.Seq;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import java.sql.Connection;
import java.sql.DriverManager;
-import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import java.util.Optional;
import java.util.Properties;
import static
[GitHub] [phoenix-connectors] stoty commented on a change in pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
stoty commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797649170
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/PhoenixDataSource.java
##
@@ -17,42 +17,76 @@
*/
package org.apache.phoenix.spark.datasource.v2;
-import java.util.Optional;
-import java.util.Properties;
-
-import org.apache.phoenix.spark.datasource.v2.reader.PhoenixDataSourceReader;
-import org.apache.phoenix.spark.datasource.v2.writer.PhoenixDataSourceWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.spark.sql.SaveMode;
+import org.apache.phoenix.spark.SparkSchemaUtil;
+import org.apache.phoenix.util.ColumnInfo;
+import org.apache.phoenix.util.PhoenixRuntime;
+import org.apache.spark.sql.connector.catalog.Table;
+import org.apache.spark.sql.connector.catalog.TableProvider;
+import org.apache.spark.sql.connector.expressions.Transform;
import org.apache.spark.sql.sources.DataSourceRegister;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.DataSourceV2;
-import org.apache.spark.sql.sources.v2.ReadSupport;
-import org.apache.spark.sql.sources.v2.WriteSupport;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
+import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
/**
* Implements the DataSourceV2 api to read and write from Phoenix tables
*/
-public class PhoenixDataSource implements DataSourceV2, ReadSupport,
WriteSupport, DataSourceRegister {
+public class PhoenixDataSource implements TableProvider, DataSourceRegister {
private static final Logger logger =
LoggerFactory.getLogger(PhoenixDataSource.class);
+public static final String TABLE_KEY = "table";
public static final String SKIP_NORMALIZING_IDENTIFIER =
"skipNormalizingIdentifier";
public static final String ZOOKEEPER_URL = "zkUrl";
public static final String PHOENIX_CONFIGS = "phoenixconfigs";
+protected StructType schema;
+private CaseInsensitiveStringMap options;
@Override
-public DataSourceReader createReader(DataSourceOptions options) {
-return new PhoenixDataSourceReader(options);
+public StructType inferSchema(CaseInsensitiveStringMap options){
+if (options.get("table") == null) {
Review comment:
use the new TABLE_KEY constant everywhere
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixScan.java
##
@@ -29,134 +29,104 @@
import org.apache.phoenix.mapreduce.PhoenixInputSplit;
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.spark.FilterExpressionCompiler;
-import org.apache.phoenix.spark.SparkSchemaUtil;
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource;
import org.apache.phoenix.util.ColumnInfo;
import org.apache.phoenix.util.PhoenixRuntime;
import org.apache.phoenix.util.QueryUtil;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.Filter;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.reader.InputPartition;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.Scan;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.PartitionReaderFactory;
import org.apache.spark.sql.types.StructType;
-import scala.Tuple3;
-import scala.collection.JavaConverters;
-import scala.collection.Seq;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import java.sql.Connection;
import java.sql.DriverManager;
-import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import java.util.Optional;
import java.util.Properties;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
-public class
[jira] [Commented] (PHOENIX-6441) Remove TSOMockModule reference from OmidTransactionProvider
[ https://issues.apache.org/jira/browse/PHOENIX-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485820#comment-17485820 ] ASF GitHub Bot commented on PHOENIX-6441: - stoty opened a new pull request #1390: URL: https://github.com/apache/phoenix/pull/1390 …der (addendum:split TransactionServiceManager to avoid CFNE with -Dwithout.tephra) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove TSOMockModule reference from OmidTransactionProvider > --- > > Key: PHOENIX-6441 > URL: https://issues.apache.org/jira/browse/PHOENIX-6441 > Project: Phoenix > Issue Type: Bug > Components: core, omid >Affects Versions: 5.2.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 4.17.0, 5.2.0, 4.16.2, 5.1.3 > > > OmidTransactionProvider references TSOMockModule which is a test dependency. > It seems to be used only by tests. > Non-test code should not depend on test artifacts. > Move the functionality into test code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [phoenix] stoty opened a new pull request #1390: PHOENIX-6441 Remove TSOMockModule reference from OmidTransactionProvi…
stoty opened a new pull request #1390: URL: https://github.com/apache/phoenix/pull/1390 …der (addendum:split TransactionServiceManager to avoid CFNE with -Dwithout.tephra) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[ https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485788#comment-17485788 ] ASF GitHub Bot commented on PHOENIX-6632: - stoty commented on pull request #69: URL: https://github.com/apache/phoenix-connectors/pull/69#issuecomment-1027933565 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 18 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 11m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 16m 55s | master passed | | +1 :green_heart: | compile | 2m 38s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | javadoc | 1m 13s | master passed | | -1 :x: | scaladoc | 0m 9s | root in master failed. | | +0 :ok: | spotbugs | 2m 12s | phoenix-spark-base in master has 40 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 43s | root in master has 116 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 9m 51s | root in the patch failed. | | -1 :x: | compile | 1m 37s | root in the patch failed. | | -1 :x: | javac | 1m 37s | root in the patch failed. | | -1 :x: | scalac | 1m 37s | root in the patch failed. | | -1 :x: | checkstyle | 0m 50s | root: The patch generated 263 new + 743 unchanged - 169 fixed = 1006 total (was 912) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | -1 :x: | javadoc | 0m 21s | phoenix-spark-base generated 180 new + 4 unchanged - 6 fixed = 184 total (was 10) | | -1 :x: | javadoc | 1m 1s | root generated 180 new + 14 unchanged - 6 fixed = 194 total (was 20) | | -1 :x: | scaladoc | 0m 38s | phoenix-spark-base in the patch failed. | | -1 :x: | scaladoc | 0m 8s | root in the patch failed. | | -1 :x: | spotbugs | 0m 40s | phoenix-spark-base in the patch failed. | | -1 :x: | spotbugs | 2m 48s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 69m 55s | root in the patch failed. | | -1 :x: | asflicense | 0m 19s | The patch generated 5 ASF License warnings. | | | | 136m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/69 | | Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs hbaseanti checkstyle scalac scaladoc | | uname | Linux 339f7c16 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev/phoenix-connectors-personality.sh | | git revision | master / 72496e5 | | Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 | | scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/branch-scaladoc-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-compile-root.txt | | scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-compile-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | javadoc |
[GitHub] [phoenix-connectors] stoty commented on pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
stoty commented on pull request #69: URL: https://github.com/apache/phoenix-connectors/pull/69#issuecomment-1027933565 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 6m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 18 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 11m 28s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 16m 55s | master passed | | +1 :green_heart: | compile | 2m 38s | master passed | | +1 :green_heart: | checkstyle | 0m 44s | master passed | | +1 :green_heart: | javadoc | 1m 13s | master passed | | -1 :x: | scaladoc | 0m 9s | root in master failed. | | +0 :ok: | spotbugs | 2m 12s | phoenix-spark-base in master has 40 extant spotbugs warnings. | | +0 :ok: | spotbugs | 3m 43s | root in master has 116 extant spotbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 9m 51s | root in the patch failed. | | -1 :x: | compile | 1m 37s | root in the patch failed. | | -1 :x: | javac | 1m 37s | root in the patch failed. | | -1 :x: | scalac | 1m 37s | root in the patch failed. | | -1 :x: | checkstyle | 0m 50s | root: The patch generated 263 new + 743 unchanged - 169 fixed = 1006 total (was 912) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | -1 :x: | javadoc | 0m 21s | phoenix-spark-base generated 180 new + 4 unchanged - 6 fixed = 184 total (was 10) | | -1 :x: | javadoc | 1m 1s | root generated 180 new + 14 unchanged - 6 fixed = 194 total (was 20) | | -1 :x: | scaladoc | 0m 38s | phoenix-spark-base in the patch failed. | | -1 :x: | scaladoc | 0m 8s | root in the patch failed. | | -1 :x: | spotbugs | 0m 40s | phoenix-spark-base in the patch failed. | | -1 :x: | spotbugs | 2m 48s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 69m 55s | root in the patch failed. | | -1 :x: | asflicense | 0m 19s | The patch generated 5 ASF License warnings. | | | | 136m 6s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/phoenix-connectors/pull/69 | | Optional Tests | dupname asflicense javac javadoc unit xml compile spotbugs hbaseanti checkstyle scalac scaladoc | | uname | Linux 339f7c16 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev/phoenix-connectors-personality.sh | | git revision | master / 72496e5 | | Default Java | Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08 | | scaladoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/branch-scaladoc-root.txt | | mvninstall | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt | | compile | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-compile-root.txt | | javac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-compile-root.txt | | scalac | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/patch-compile-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | javadoc | https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-Connectors-PreCommit-GitHub-PR/job/PR-69/2/artifact/yetus-general-check/output/diff-javadoc-javadoc-phoenix-spark-base.txt | | javadoc |
[jira] [Commented] (PHOENIX-5865) column that has default value can not be correctly indexed
[
https://issues.apache.org/jira/browse/PHOENIX-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485759#comment-17485759
]
Richárd Antal commented on PHOENIX-5865:
Thanks [~stoty] for the reminder, I pushed to the 4.x and 4.16 branches now.
> column that has default value can not be correctly indexed
> --
>
> Key: PHOENIX-5865
> URL: https://issues.apache.org/jira/browse/PHOENIX-5865
> Project: Phoenix
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: junfei liang
>Assignee: Richárd Antal
>Priority: Major
> Fix For: 5.2.0, 5.1.3
>
> Attachments: IndexWithDefaultIT.patch
>
>
> # create a table with a column has default value
> create table data_table(
> pk VARCHAR, +
> c VARCHAR default '0',
> CONSTRAINT my_pk PRIMARY KEY (pk)
> 2. insert data into it
> upsert into data_table values('1','1')
> 3. create an index on the column
> CREATE INDEX idx_data_table ON data_table(pk, c)
> 4. check index table ,found data is wrong
> select * from idx_data_table
> found 0:c column value is '0' , but it should be 1.
> see attachment for details.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (PHOENIX-5865) column that has default value can not be correctly indexed
[
https://issues.apache.org/jira/browse/PHOENIX-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485755#comment-17485755
]
ASF GitHub Bot commented on PHOENIX-5865:
-
richardantal merged pull request #1389:
URL: https://github.com/apache/phoenix/pull/1389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> column that has default value can not be correctly indexed
> --
>
> Key: PHOENIX-5865
> URL: https://issues.apache.org/jira/browse/PHOENIX-5865
> Project: Phoenix
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: junfei liang
>Assignee: Richárd Antal
>Priority: Major
> Fix For: 5.2.0, 5.1.3
>
> Attachments: IndexWithDefaultIT.patch
>
>
> # create a table with a column has default value
> create table data_table(
> pk VARCHAR, +
> c VARCHAR default '0',
> CONSTRAINT my_pk PRIMARY KEY (pk)
> 2. insert data into it
> upsert into data_table values('1','1')
> 3. create an index on the column
> CREATE INDEX idx_data_table ON data_table(pk, c)
> 4. check index table ,found data is wrong
> select * from idx_data_table
> found 0:c column value is '0' , but it should be 1.
> see attachment for details.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] richardantal merged pull request #1389: PHOENIX-5865 Column that has default value can not be correctly indexed
richardantal merged pull request #1389: URL: https://github.com/apache/phoenix/pull/1389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[
https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485683#comment-17485683
]
ASF GitHub Bot commented on PHOENIX-6632:
-
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797456009
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReadOptions.java
##
@@ -57,6 +59,14 @@ String getTenantId() {
}
Properties getOverriddenProps() {
Review comment:
I have modified the way this was implemented, I went back to how it is
currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Migrate connectors to Spark-3
> -
>
> Key: PHOENIX-6632
> URL: https://issues.apache.org/jira/browse/PHOENIX-6632
> Project: Phoenix
> Issue Type: Improvement
> Components: spark-connector
>Affects Versions: connectors-6.0.0
>Reporter: Ashwin Balasubramani
>Priority: Major
>
> With Spark-3, the DatasourceV2 API has had major changes, where a new
> TableProvider Interface has been introduced. These new changes bring in more
> control to the data source developer and better integration with
> spark-optimizer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[
https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485684#comment-17485684
]
ASF GitHub Bot commented on PHOENIX-6632:
-
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797456215
##
File path: phoenix-spark-base/pom.xml
##
@@ -121,7 +121,7 @@
org.scalatest
scalatest_${scala.binary.version}
- 2.2.4
+ 3.1.0
Review comment:
No specific reason, I can change this to latest version if you'd think
that would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Migrate connectors to Spark-3
> -
>
> Key: PHOENIX-6632
> URL: https://issues.apache.org/jira/browse/PHOENIX-6632
> Project: Phoenix
> Issue Type: Improvement
> Components: spark-connector
>Affects Versions: connectors-6.0.0
>Reporter: Ashwin Balasubramani
>Priority: Major
>
> With Spark-3, the DatasourceV2 API has had major changes, where a new
> TableProvider Interface has been introduced. These new changes bring in more
> control to the data source developer and better integration with
> spark-optimizer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix-connectors] ashwinb1998 commented on a change in pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797456215
##
File path: phoenix-spark-base/pom.xml
##
@@ -121,7 +121,7 @@
org.scalatest
scalatest_${scala.binary.version}
- 2.2.4
+ 3.1.0
Review comment:
No specific reason, I can change this to latest version if you'd think
that would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [phoenix-connectors] ashwinb1998 commented on a change in pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797456009
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixDataSourceReadOptions.java
##
@@ -57,6 +59,14 @@ String getTenantId() {
}
Properties getOverriddenProps() {
Review comment:
I have modified the way this was implemented, I went back to how it is
currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[
https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485681#comment-17485681
]
ASF GitHub Bot commented on PHOENIX-6632:
-
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797455273
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/writer/PhoenixBatchWrite.java
##
@@ -17,45 +17,35 @@
*/
package org.apache.phoenix.spark.datasource.v2.writer;
-import org.apache.phoenix.util.PhoenixRuntime;
-import org.apache.spark.sql.SaveMode;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
-import org.apache.spark.sql.sources.v2.writer.DataWriterFactory;
-import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage;
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.spark.sql.connector.write.BatchWrite;
+import org.apache.spark.sql.connector.write.DataWriterFactory;
+import org.apache.spark.sql.connector.write.LogicalWriteInfo;
+import org.apache.spark.sql.connector.write.PhysicalWriteInfo;
+import org.apache.spark.sql.connector.write.WriterCommitMessage;
import org.apache.spark.sql.types.StructType;
+import java.util.Map;
+
import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.CURRENT_SCN_VALUE;
-import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
+import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.ZOOKEEPER_URL;
+import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
-public class PhoenixDataSourceWriter implements DataSourceWriter {
+public class PhoenixBatchWrite implements BatchWrite {
private final PhoenixDataSourceWriteOptions options;
+private final LogicalWriteInfo writeInfo;
-public PhoenixDataSourceWriter(SaveMode mode, StructType schema,
DataSourceOptions options) {
-if (!mode.equals(SaveMode.Overwrite)) {
-throw new RuntimeException("SaveMode other than SaveMode.OverWrite
is not supported");
-}
-if (!options.tableName().isPresent()) {
-throw new RuntimeException("No Phoenix option " +
DataSourceOptions.TABLE_KEY + " defined");
-}
-if (!options.get(ZOOKEEPER_URL).isPresent()) {
-throw new RuntimeException("No Phoenix option " + ZOOKEEPER_URL +
" defined");
-}
-this.options = createPhoenixDataSourceWriteOptions(options, schema);
-}
-
-@Override
-public DataWriterFactory createWriterFactory() {
-return new PhoenixDataWriterFactory(options);
+PhoenixBatchWrite(LogicalWriteInfo writeInfo, Map options) {
+this.writeInfo = writeInfo;
+this.options = createPhoenixDataSourceWriteOptions(options,
writeInfo.schema());
}
@Override
-public boolean useCommitCoordinator() {
Review comment:
It basically returns whether Spark should use the commit coordinator to
ensure that at most one task for each partition commits. It has now been
removed with DSV2 in spark3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Migrate connectors to Spark-3
> -
>
> Key: PHOENIX-6632
> URL: https://issues.apache.org/jira/browse/PHOENIX-6632
> Project: Phoenix
> Issue Type: Improvement
> Components: spark-connector
>Affects Versions: connectors-6.0.0
>Reporter: Ashwin Balasubramani
>Priority: Major
>
> With Spark-3, the DatasourceV2 API has had major changes, where a new
> TableProvider Interface has been introduced. These new changes bring in more
> control to the data source developer and better integration with
> spark-optimizer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (PHOENIX-6632) Migrate connectors to Spark-3
[
https://issues.apache.org/jira/browse/PHOENIX-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485682#comment-17485682
]
ASF GitHub Bot commented on PHOENIX-6632:
-
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797455447
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/writer/PhoenixBatchWrite.java
##
@@ -17,45 +17,35 @@
*/
package org.apache.phoenix.spark.datasource.v2.writer;
-import org.apache.phoenix.util.PhoenixRuntime;
-import org.apache.spark.sql.SaveMode;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
-import org.apache.spark.sql.sources.v2.writer.DataWriterFactory;
-import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage;
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.spark.sql.connector.write.BatchWrite;
+import org.apache.spark.sql.connector.write.DataWriterFactory;
+import org.apache.spark.sql.connector.write.LogicalWriteInfo;
+import org.apache.spark.sql.connector.write.PhysicalWriteInfo;
+import org.apache.spark.sql.connector.write.WriterCommitMessage;
import org.apache.spark.sql.types.StructType;
+import java.util.Map;
+
import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.CURRENT_SCN_VALUE;
-import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
+import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.ZOOKEEPER_URL;
+import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
-public class PhoenixDataSourceWriter implements DataSourceWriter {
+public class PhoenixBatchWrite implements BatchWrite {
private final PhoenixDataSourceWriteOptions options;
+private final LogicalWriteInfo writeInfo;
-public PhoenixDataSourceWriter(SaveMode mode, StructType schema,
DataSourceOptions options) {
-if (!mode.equals(SaveMode.Overwrite)) {
Review comment:
SaveMode has been removed from the DataSourceV2 API and hence the there
is no parameter SaveMode that can be used here. Reference:
https://issues.apache.org/jira/browse/SPARK-25531
It is a requirement though, that for the writes, SaveMode should be
"Append".
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixScan.java
##
@@ -29,134 +29,97 @@
import org.apache.phoenix.mapreduce.PhoenixInputSplit;
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.spark.FilterExpressionCompiler;
-import org.apache.phoenix.spark.SparkSchemaUtil;
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource;
import org.apache.phoenix.util.ColumnInfo;
import org.apache.phoenix.util.PhoenixRuntime;
import org.apache.phoenix.util.QueryUtil;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.Filter;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.reader.InputPartition;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.Scan;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.PartitionReaderFactory;
import org.apache.spark.sql.types.StructType;
-import scala.Tuple3;
-import scala.collection.JavaConverters;
-import scala.collection.Seq;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import java.sql.Connection;
import java.sql.DriverManager;
-import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import java.util.Optional;
import java.util.Properties;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
-public class PhoenixDataSourceReader implements DataSourceReader,
SupportsPushDownFilters,
-SupportsPushDownRequiredColumns {
+public class PhoenixScan implements Scan, Batch {
-private final DataSourceOptions options;
-
[GitHub] [phoenix-connectors] ashwinb1998 commented on a change in pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797455447
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/writer/PhoenixBatchWrite.java
##
@@ -17,45 +17,35 @@
*/
package org.apache.phoenix.spark.datasource.v2.writer;
-import org.apache.phoenix.util.PhoenixRuntime;
-import org.apache.spark.sql.SaveMode;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
-import org.apache.spark.sql.sources.v2.writer.DataWriterFactory;
-import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage;
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.spark.sql.connector.write.BatchWrite;
+import org.apache.spark.sql.connector.write.DataWriterFactory;
+import org.apache.spark.sql.connector.write.LogicalWriteInfo;
+import org.apache.spark.sql.connector.write.PhysicalWriteInfo;
+import org.apache.spark.sql.connector.write.WriterCommitMessage;
import org.apache.spark.sql.types.StructType;
+import java.util.Map;
+
import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.CURRENT_SCN_VALUE;
-import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
+import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.ZOOKEEPER_URL;
+import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
-public class PhoenixDataSourceWriter implements DataSourceWriter {
+public class PhoenixBatchWrite implements BatchWrite {
private final PhoenixDataSourceWriteOptions options;
+private final LogicalWriteInfo writeInfo;
-public PhoenixDataSourceWriter(SaveMode mode, StructType schema,
DataSourceOptions options) {
-if (!mode.equals(SaveMode.Overwrite)) {
Review comment:
SaveMode has been removed from the DataSourceV2 API and hence the there
is no parameter SaveMode that can be used here. Reference:
https://issues.apache.org/jira/browse/SPARK-25531
It is a requirement though, that for the writes, SaveMode should be
"Append".
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/reader/PhoenixScan.java
##
@@ -29,134 +29,97 @@
import org.apache.phoenix.mapreduce.PhoenixInputSplit;
import org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil;
import org.apache.phoenix.query.KeyRange;
-import org.apache.phoenix.spark.FilterExpressionCompiler;
-import org.apache.phoenix.spark.SparkSchemaUtil;
import org.apache.phoenix.spark.datasource.v2.PhoenixDataSource;
import org.apache.phoenix.util.ColumnInfo;
import org.apache.phoenix.util.PhoenixRuntime;
import org.apache.phoenix.util.QueryUtil;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.Filter;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.reader.DataSourceReader;
-import org.apache.spark.sql.sources.v2.reader.InputPartition;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters;
-import org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns;
+import org.apache.spark.sql.connector.read.Batch;
+import org.apache.spark.sql.connector.read.Scan;
+import org.apache.spark.sql.connector.read.InputPartition;
+import org.apache.spark.sql.connector.read.PartitionReaderFactory;
import org.apache.spark.sql.types.StructType;
-import scala.Tuple3;
-import scala.collection.JavaConverters;
-import scala.collection.Seq;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
import java.sql.Connection;
import java.sql.DriverManager;
-import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import java.util.Optional;
import java.util.Properties;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL;
import static org.apache.phoenix.util.PhoenixRuntime.JDBC_PROTOCOL_SEPARATOR;
-public class PhoenixDataSourceReader implements DataSourceReader,
SupportsPushDownFilters,
-SupportsPushDownRequiredColumns {
+public class PhoenixScan implements Scan, Batch {
-private final DataSourceOptions options;
-private final String tableName;
+private final StructType schema;
+private final CaseInsensitiveStringMap options;
private final String zkUrl;
-private final boolean dateAsTimestamp;
private final Properties overriddenProps;
+
[GitHub] [phoenix-connectors] ashwinb1998 commented on a change in pull request #69: PHOENIX-6632 Migrate/Update connectors to spark-3
ashwinb1998 commented on a change in pull request #69:
URL: https://github.com/apache/phoenix-connectors/pull/69#discussion_r797455273
##
File path:
phoenix-spark-base/src/main/java/org/apache/phoenix/spark/datasource/v2/writer/PhoenixBatchWrite.java
##
@@ -17,45 +17,35 @@
*/
package org.apache.phoenix.spark.datasource.v2.writer;
-import org.apache.phoenix.util.PhoenixRuntime;
-import org.apache.spark.sql.SaveMode;
-import org.apache.spark.sql.catalyst.InternalRow;
-import org.apache.spark.sql.sources.v2.DataSourceOptions;
-import org.apache.spark.sql.sources.v2.writer.DataSourceWriter;
-import org.apache.spark.sql.sources.v2.writer.DataWriterFactory;
-import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage;
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.spark.sql.connector.write.BatchWrite;
+import org.apache.spark.sql.connector.write.DataWriterFactory;
+import org.apache.spark.sql.connector.write.LogicalWriteInfo;
+import org.apache.spark.sql.connector.write.PhysicalWriteInfo;
+import org.apache.spark.sql.connector.write.WriterCommitMessage;
import org.apache.spark.sql.types.StructType;
+import java.util.Map;
+
import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.CURRENT_SCN_VALUE;
-import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
+import static
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.MAPREDUCE_TENANT_ID;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.ZOOKEEPER_URL;
+import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.SKIP_NORMALIZING_IDENTIFIER;
import static
org.apache.phoenix.spark.datasource.v2.PhoenixDataSource.extractPhoenixHBaseConfFromOptions;
-public class PhoenixDataSourceWriter implements DataSourceWriter {
+public class PhoenixBatchWrite implements BatchWrite {
private final PhoenixDataSourceWriteOptions options;
+private final LogicalWriteInfo writeInfo;
-public PhoenixDataSourceWriter(SaveMode mode, StructType schema,
DataSourceOptions options) {
-if (!mode.equals(SaveMode.Overwrite)) {
-throw new RuntimeException("SaveMode other than SaveMode.OverWrite
is not supported");
-}
-if (!options.tableName().isPresent()) {
-throw new RuntimeException("No Phoenix option " +
DataSourceOptions.TABLE_KEY + " defined");
-}
-if (!options.get(ZOOKEEPER_URL).isPresent()) {
-throw new RuntimeException("No Phoenix option " + ZOOKEEPER_URL +
" defined");
-}
-this.options = createPhoenixDataSourceWriteOptions(options, schema);
-}
-
-@Override
-public DataWriterFactory createWriterFactory() {
-return new PhoenixDataWriterFactory(options);
+PhoenixBatchWrite(LogicalWriteInfo writeInfo, Map options) {
+this.writeInfo = writeInfo;
+this.options = createPhoenixDataSourceWriteOptions(options,
writeInfo.schema());
}
@Override
-public boolean useCommitCoordinator() {
Review comment:
It basically returns whether Spark should use the commit coordinator to
ensure that at most one task for each partition commits. It has now been
removed with DSV2 in spark3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (PHOENIX-5865) column that has default value can not be correctly indexed
[
https://issues.apache.org/jira/browse/PHOENIX-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485630#comment-17485630
]
ASF GitHub Bot commented on PHOENIX-5865:
-
richardantal closed pull request #1372:
URL: https://github.com/apache/phoenix/pull/1372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> column that has default value can not be correctly indexed
> --
>
> Key: PHOENIX-5865
> URL: https://issues.apache.org/jira/browse/PHOENIX-5865
> Project: Phoenix
> Issue Type: Bug
> Components: core
>Affects Versions: 5.0.0
>Reporter: junfei liang
>Assignee: Richárd Antal
>Priority: Major
> Fix For: 5.2.0, 5.1.3
>
> Attachments: IndexWithDefaultIT.patch
>
>
> # create a table with a column has default value
> create table data_table(
> pk VARCHAR, +
> c VARCHAR default '0',
> CONSTRAINT my_pk PRIMARY KEY (pk)
> 2. insert data into it
> upsert into data_table values('1','1')
> 3. create an index on the column
> CREATE INDEX idx_data_table ON data_table(pk, c)
> 4. check index table ,found data is wrong
> select * from idx_data_table
> found 0:c column value is '0' , but it should be 1.
> see attachment for details.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [phoenix] richardantal closed pull request #1372: PHOENIX-5865 Column that has default value can not be correctly indexed
richardantal closed pull request #1372: URL: https://github.com/apache/phoenix/pull/1372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@phoenix.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (PHOENIX-5894) Table versus Table Full Outer join on Salted tables not working
[ https://issues.apache.org/jira/browse/PHOENIX-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485624#comment-17485624 ] Sergey Soldatov edited comment on PHOENIX-5894 at 2/2/22, 8:35 AM: --- Just some thoughts. Whether it's fine that the projected table has columns with the column type different from ProjectedColumn? It didn't happen earlier, so it might be the case that we broke the logic somewhere. It seems that this problem is a regression for PHOENIX-4508. It implicitly adds the number of salt buckets to the projected ptable regardless of whether retainPKColumns is set or not. was (Author: sergey.soldatov): Just some thoughts. Whether it's fine that the projected table has columns with the column type different from ProjectedColumn? It didn't happen earlier, so it might be the case that we broke the logic somewhere. > Table versus Table Full Outer join on Salted tables not working > --- > > Key: PHOENIX-5894 > URL: https://issues.apache.org/jira/browse/PHOENIX-5894 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.0.0 >Reporter: Ben Cohen >Assignee: Rajeshbabu Chintaguntla >Priority: Major > Fix For: 4.17.0, 4.16.2 > > Attachments: PHOENIX-5894.patch, > Salt_Bucketed_Table_Full_Outer_Join_Issue.docx > > > Trying to do a Full Outer Join on two tables. The operation works when the > tables are not salted but fails with an exception related to casting when > performed on the salted versions of the tables. Here is the exceptions: > "java.lang.ClassCastException: org.apache.phoenix.schema.PColumnImpl cannot > be cast to org.apache.phoenix.schema.ProjectedColumn > at > org.apache.phoenix.compile.JoinCompiler.joinProjectedTables(JoinCompiler.java:1256) > at > org.apache.phoenix.compile.QueryCompiler.compileJoinQuery(QueryCompiler.java:425) > at > org.apache.phoenix.compile.QueryCompiler.compileJoinQuery(QueryCompiler.java:228) > at > org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:191) > at > org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:153) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:490) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:456) > at > org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:302) > at > org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:291) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:290) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:283) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1830) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:813) > at sqlline.SqlLine.begin(SqlLine.java:686) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:291)" > I have attached a word document with the complete list of queries and their > results, along with commands to recreate the data. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
