[jira] [Commented] (PHOENIX-2084) load json data from rabbitmq using apache phoenix flume plugin into HBase
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395497#comment-15395497 ] Kalyan commented on PHOENIX-2084: - I just provided the solution for JSON data with flume. please verify the solution patch: https://github.com/kalyanhadooptraining/phoenix/commit/501cc2eb0a289d3f29be4daec28cc217ef62917e any suggestions .. please update me. > load json data from rabbitmq using apache phoenix flume plugin into HBase > - > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: Task > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda > > I am trying load j son data from rabbitmq using apace phoenix flume plugin > into HBase but the flume supports only regular expression > Is there any best solution to load the json data into HBase using Apache > phoenix flume. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400327#comment-15400327 ] Kalyan commented on PHOENIX-2084: - Hi Josh Mahonin , i will work on Array Columns support .. i will update when the patch is ready. I am also planning to work on csv/tsv data to move to phoenix table using flume (instead of providing complex regex for csv , tsv data .. by using csv parser) Thanks for your comments > Support loading json data using apache phoenix flume plugin > --- > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda >Assignee: venakt rao sirikonda > Fix For: 4.9.0 > > > We should support loading JSON data through our flume plugin as flume as > support for this data shape. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402064#comment-15402064
]
Kalyan commented on PHOENIX-2084:
-
as per previous comments .. i updated the code .
now code will work with array data type also.
Please verify the patch .. i included the test cases also.
https://github.com/kalyanhadooptraining/phoenix/commit/fd9cbf36dc814fbce087626c8fff909222fbe6f9
It supports array data type also with proper `jsonpath`
Note:
// mandatory properties
phoenix-agent.sinks.phoenixsink.serializer = json
phoenix-agent.sinks.phoenixsink.serializer.columns=c1,c2,c3,c4
// optional properties with sample data
phoenix-agent.sinks.phoenixsink.serializer.partialSchema = true
// {"c1":"kalyan", "c2":1, "c3":2012, "c4":"phoenix"}
phoenix-agent.sinks.phoenixsink.serializer.columnsMapping = {"c1":"c1",
"c2":"c2", "c3":"c3", "c4":"c4"}
// {"f1":"kalyan", "f2":1, "f3":2012, "f4":"phoenix"}
phoenix-agent.sinks.phoenixsink.serializer.columnsMapping = {"c1":"f1",
"c2":"f2", "c3":"f3", "c4":"f4"}
// {"c1":"kalyan", "c2":1, "a": {"b":2012}, "x":{"y":"phoenix"}}
phoenix-agent.sinks.phoenixsink.serializer.columnsMapping = {"c1":"c1",
"c2":"c2", "c3":"a.b", "c4":"x.y"}
// {"c1":"kalyan", "c2":1, "a": {"b":2012}, "x":{"y": [{"z" : "phoenix"}, {"z"
: "flume"}] }}
phoenix-agent.sinks.phoenixsink.serializer.columnsMapping = {"c1":"c1",
"c2":"c2", "c3":"a.b", "c4":"x.y[*].z"}
any suggestions .. please update me.
> Support loading json data using apache phoenix flume plugin
> ---
>
> Key: PHOENIX-2084
> URL: https://issues.apache.org/jira/browse/PHOENIX-2084
> Project: Phoenix
> Issue Type: New Feature
> Environment: cloudera 5.4
>Reporter: venakt rao sirikonda
>Assignee: venakt rao sirikonda
> Fix For: 4.9.0
>
>
> We should support loading JSON data through our flume plugin as flume as
> support for this data shape.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
Kalyan created PHOENIX-3135: --- Summary: Support loading csv data using apache phoenix flume plugin Key: PHOENIX-3135 URL: https://issues.apache.org/jira/browse/PHOENIX-3135 Project: Phoenix Issue Type: New Feature Environment: cloudera 5.4 Reporter: Kalyan Priority: Minor Fix For: 4.9.0 To work with below sample data sets ... we need support loading csv data using apache phoenix flume plugin. // sample data set 1 schema: col1 varchar , col2 double, col3 varchar, col4 integer input: kalyan,10.5,abc,1 input: "kalyan",10.5,"abc",1 // sample data set 1 schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403529#comment-15403529 ] Kalyan commented on PHOENIX-3135: - I just provided the solution for CSV data with flume. please verify the solution patch: https://github.com/kalyanhadooptraining/phoenix/commit/04234ce7564326da564898c3c726640db23d9c66 Adding new feature to existing PhoenixSink. Convert CSV Data to Phoenix Table Data It supports array data type also with proper `array delimiter` Note: // mandatory properties phoenix-agent.sinks.phoenixsink.serializer = csv phoenix-agent.sinks.phoenixsink.serializer.columns=c1,c2,c3,c4 // optional properties with sample data phoenix-agent.sinks.phoenixsink.serializer.csvDelimiter = , phoenix-agent.sinks.phoenixsink.serializer.csvQuote = \" phoenix-agent.sinks.phoenixsink.serializer.csvEscape = \\ phoenix-agent.sinks.phoenixsink.serializer.csvArrayDelimiter = , // sample data set 1 schema: col1 varchar , col2 double, col3 varchar, col4 integer input: kalyan,10.5,abc,1 input: "kalyan",10.5,"abc",1 // sample data set 1 schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" any suggestions .. please update me. > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Priority: Minor > Fix For: 4.9.0 > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3135: Attachment: phoenix_csv.patch Patch is available for "Support loading csv data using apache phoenix flume plugin" > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Priority: Minor > Fix For: 4.9.0 > > Attachments: phoenix_csv.patch > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-2084: Attachment: phoenix_json.patch Patch is available for "Support loading json data using apache phoenix flume plugin" > Support loading json data using apache phoenix flume plugin > --- > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda >Assignee: venakt rao sirikonda > Fix For: 4.9.0 > > Attachments: phoenix_json.patch > > > We should support loading JSON data through our flume plugin as flume as > support for this data shape. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2547) Spark Data Source API: Filter operation doesn't work for column names containing a white space
[
https://issues.apache.org/jira/browse/PHOENIX-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404113#comment-15404113
]
Kalyan commented on PHOENIX-2547:
-
Provided the solution for above bug & written the test case
Please verify the patch .. i included the test cases also.
https://github.com/kalyanhadooptraining/phoenix/commit/937cab227c26bc364129e6395bf06378ee536103
Verify the solution:
1. Create a test table & insert a row as below
create table "space" ("key" varchar primary key, "first name" varchar);
upsert into "space" values ('key1', 'xyz');
2. Verify in spark with below queries
val df = sqlContext.load("org.apache.phoenix.spark", Map("table" ->
"\"space\"", "zkUrl" -> "localhost:2181"))
val filterdf = df.filter(df.col("first name").equalTo("xyz"))
filterdf.collect
> Spark Data Source API: Filter operation doesn't work for column names
> containing a white space
> --
>
> Key: PHOENIX-2547
> URL: https://issues.apache.org/jira/browse/PHOENIX-2547
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
>Priority: Critical
> Labels: verify
> Fix For: 4.9.0
>
>
> Dataframe.filter() results in
> "org.apache.phoenix.exception.PhoenixParserException: ERROR 604 (42P00):
> Syntax error. Mismatched input. Expecting "LPAREN", got "first" at line 1,
> column 52." when a column name has a white space in it.
> Steps to Reproduce
> --
> 1. Create a test table & insert a row as below
>create table "space" ("key" varchar primary key, "first name" varchar);
>upsert into "space" values ('key1', 'xyz');
> 2. Java code that leads to the error:
> //omitting the DataFrame creation part
>df = df.filter(df.col("first name").equalTo("xyz"));
> System.out.println(df.collectAsList());
> 3. I could see the following statements in the Phoenix logs which may have
> led to the exception (stack trace given below)
> 2015-12-28 17:52:24,327 INFO [main]
> org.apache.phoenix.mapreduce.PhoenixInputFormat
> UseSelectColumns=true, selectColumnList.size()=2, selectColumnList=key,first
> name
> 2015-12-28 17:52:24,328 INFO [main]
> org.apache.phoenix.mapreduce.PhoenixInputFormat
> Select Statement: SELECT "key","0"."first name" FROM "space" WHERE ( first
> name = 'xyz')
> 2015-12-28 17:52:24,333 ERROR [main]
> org.apache.phoenix.mapreduce.PhoenixInputFormat
> Failed to get the query plan with error [ERROR 604 (42P00): Syntax error.
> Mismatched input. Expecting "LPAREN", got "first" at line 1, column 52.]
> Exception Stack Trace:
> --
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixParserException: ERROR 604 (42P00):
> Syntax error. Mismatched input. Expecting "LPAREN", got "first" at line 1,
> column 52.
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.phoenix.spark.PhoenixRDD.getPartitions(PhoenixRDD.scala:48)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scal
[jira] [Updated] (PHOENIX-2547) Spark Data Source API: Filter operation doesn't work for column names containing a white space
[
https://issues.apache.org/jira/browse/PHOENIX-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kalyan updated PHOENIX-2547:
Attachment: phoenix_spark.patch
Patch is available for "Spark Data Source API: Filter operation doesn't work
for column names containing a white space"..
> Spark Data Source API: Filter operation doesn't work for column names
> containing a white space
> --
>
> Key: PHOENIX-2547
> URL: https://issues.apache.org/jira/browse/PHOENIX-2547
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
>Priority: Critical
> Labels: verify
> Fix For: 4.9.0
>
> Attachments: phoenix_spark.patch
>
>
> Dataframe.filter() results in
> "org.apache.phoenix.exception.PhoenixParserException: ERROR 604 (42P00):
> Syntax error. Mismatched input. Expecting "LPAREN", got "first" at line 1,
> column 52." when a column name has a white space in it.
> Steps to Reproduce
> --
> 1. Create a test table & insert a row as below
>create table "space" ("key" varchar primary key, "first name" varchar);
>upsert into "space" values ('key1', 'xyz');
> 2. Java code that leads to the error:
> //omitting the DataFrame creation part
>df = df.filter(df.col("first name").equalTo("xyz"));
> System.out.println(df.collectAsList());
> 3. I could see the following statements in the Phoenix logs which may have
> led to the exception (stack trace given below)
> 2015-12-28 17:52:24,327 INFO [main]
> org.apache.phoenix.mapreduce.PhoenixInputFormat
> UseSelectColumns=true, selectColumnList.size()=2, selectColumnList=key,first
> name
> 2015-12-28 17:52:24,328 INFO [main]
> org.apache.phoenix.mapreduce.PhoenixInputFormat
> Select Statement: SELECT "key","0"."first name" FROM "space" WHERE ( first
> name = 'xyz')
> 2015-12-28 17:52:24,333 ERROR [main]
> org.apache.phoenix.mapreduce.PhoenixInputFormat
> Failed to get the query plan with error [ERROR 604 (42P00): Syntax error.
> Mismatched input. Expecting "LPAREN", got "first" at line 1, column 52.]
> Exception Stack Trace:
> --
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixParserException: ERROR 604 (42P00):
> Syntax error. Mismatched input. Expecting "LPAREN", got "first" at line 1,
> column 52.
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.phoenix.spark.PhoenixRDD.getPartitions(PhoenixRDD.scala:48)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
[jira] [Commented] (PHOENIX-3074) Phoenix Bulkload fails when Table name contains '-' (hyphen)
[
https://issues.apache.org/jira/browse/PHOENIX-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404593#comment-15404593
]
Kalyan commented on PHOENIX-3074:
-
Provided the solution for above bug & written the test case
Please verify the patch .. i included the test cases also.
https://github.com/kalyanhadooptraining/phoenix/commit/19e999c96594370423c5912ec41896f9b7ff8344
> Phoenix Bulkload fails when Table name contains '-' (hyphen)
>
>
> Key: PHOENIX-3074
> URL: https://issues.apache.org/jira/browse/PHOENIX-3074
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Dharmesh
> Labels: easyfix, features
> Fix For: 4.7.0
>
>
> Bulk load fails from csv to Phoenix table when Phoenix table contains ‘-‘
> (hyphen) in table name. [Phoenix 4.7.0]
> Below is the steps to reproduce the issue.
> Step-1: create table with '-' hyphen. Use double quote around table name.
> CREATE TABLE "PHOENIX-TEST" (ID INTEGER PRIMARY KEY, NAME VARCHAR);
> Step-2:
> sudo -u hadoop
> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-protocol.jar:/usr/lib/hbase/conf/
> hadoop jar /usr/lib/phoenix/phoenix-client.jar
> org.apache.phoenix.mapreduce.CsvBulkLoadTool -Dfs.permissions.umask-mode=000
> -t "PHOENIX-TEST" --input "/user/test/PHOENIX-TEST.csv" -d $'\t'
> Below is the stack trace for the error. Ultimately it removes any double or
> single quotes around table name from the bulk load command.
> 2016-07-14 10:20:47,153 WARN [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> already initialized!
> 2016-07-14 10:20:48,703 INFO [main] org.apache.phoenix.util.UpsertExecutor:
> Upserting SQL data with UPSERT INTO PHOENIX-TEST ("ID", "0"."NAME") VALUES
> (?, ?)
> 2016-07-14 10:20:48,704 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2016-07-14 10:20:48,711 INFO [main] org.apache.hadoop.io.compress.CodecPool:
> Got brand-new compressor [.snappy]
> 2016-07-14 10:20:48,718 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00):
> Syntax error. Encountered "-" at line 1, column 17.
> at
> org.apache.phoenix.util.UpsertExecutor.createStatement(UpsertExecutor.java:83)
> at org.apache.phoenix.util.UpsertExecutor.(UpsertExecutor.java:94)
> at
> org.apache.phoenix.util.csv.CsvUpsertExecutor.(CsvUpsertExecutor.java:63)
> at
> org.apache.phoenix.mapreduce.CsvToKeyValueMapper.buildUpsertExecutor(CsvToKeyValueMapper.java:85)
> at
> org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.setup(FormatToBytesWritableMapper.java:142)
> at
> org.apache.phoenix.mapreduce.CsvToKeyValueMapper.setup(CsvToKeyValueMapper.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.phoenix.exception.PhoenixParserException: ERROR 601
> (42P00): Syntax error. Encountered "-" at line 1, column 17.
> at
> org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1185)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1268)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.(PhoenixPreparedStatement.java:94)
> at
> org.apache.phoenix.jdbc.PhoenixConnection.prepareStatement(PhoenixConnection.java:715)
> at
> org.apache.phoenix.util.UpsertExecutor.createStatement(UpsertExecutor.java:81)
> ... 13 more
> Caused by: NoViableAltException(94@[])
> at
> org.apache.phoenix.parse.PhoenixSQLParser.upsert_node(PhoenixSQLParser.java:4723)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:775)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:500)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108)
> ... 18 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-3074) Phoenix Bulkload fails when Table name contains '-' (hyphen)
[
https://issues.apache.org/jira/browse/PHOENIX-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kalyan updated PHOENIX-3074:
Attachment: phoenix_hyphen.patch
Patch is available for "Phoenix Bulkload fails when Table name contains '-'
(hyphen)"..
> Phoenix Bulkload fails when Table name contains '-' (hyphen)
>
>
> Key: PHOENIX-3074
> URL: https://issues.apache.org/jira/browse/PHOENIX-3074
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Dharmesh
> Labels: easyfix, features
> Fix For: 4.7.0
>
> Attachments: phoenix_hyphen.patch
>
>
> Bulk load fails from csv to Phoenix table when Phoenix table contains ‘-‘
> (hyphen) in table name. [Phoenix 4.7.0]
> Below is the steps to reproduce the issue.
> Step-1: create table with '-' hyphen. Use double quote around table name.
> CREATE TABLE "PHOENIX-TEST" (ID INTEGER PRIMARY KEY, NAME VARCHAR);
> Step-2:
> sudo -u hadoop
> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-protocol.jar:/usr/lib/hbase/conf/
> hadoop jar /usr/lib/phoenix/phoenix-client.jar
> org.apache.phoenix.mapreduce.CsvBulkLoadTool -Dfs.permissions.umask-mode=000
> -t "PHOENIX-TEST" --input "/user/test/PHOENIX-TEST.csv" -d $'\t'
> Below is the stack trace for the error. Ultimately it removes any double or
> single quotes around table name from the bulk load command.
> 2016-07-14 10:20:47,153 WARN [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> already initialized!
> 2016-07-14 10:20:48,703 INFO [main] org.apache.phoenix.util.UpsertExecutor:
> Upserting SQL data with UPSERT INTO PHOENIX-TEST ("ID", "0"."NAME") VALUES
> (?, ?)
> 2016-07-14 10:20:48,704 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2016-07-14 10:20:48,711 INFO [main] org.apache.hadoop.io.compress.CodecPool:
> Got brand-new compressor [.snappy]
> 2016-07-14 10:20:48,718 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00):
> Syntax error. Encountered "-" at line 1, column 17.
> at
> org.apache.phoenix.util.UpsertExecutor.createStatement(UpsertExecutor.java:83)
> at org.apache.phoenix.util.UpsertExecutor.(UpsertExecutor.java:94)
> at
> org.apache.phoenix.util.csv.CsvUpsertExecutor.(CsvUpsertExecutor.java:63)
> at
> org.apache.phoenix.mapreduce.CsvToKeyValueMapper.buildUpsertExecutor(CsvToKeyValueMapper.java:85)
> at
> org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.setup(FormatToBytesWritableMapper.java:142)
> at
> org.apache.phoenix.mapreduce.CsvToKeyValueMapper.setup(CsvToKeyValueMapper.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.phoenix.exception.PhoenixParserException: ERROR 601
> (42P00): Syntax error. Encountered "-" at line 1, column 17.
> at
> org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1185)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1268)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.(PhoenixPreparedStatement.java:94)
> at
> org.apache.phoenix.jdbc.PhoenixConnection.prepareStatement(PhoenixConnection.java:715)
> at
> org.apache.phoenix.util.UpsertExecutor.createStatement(UpsertExecutor.java:81)
> ... 13 more
> Caused by: NoViableAltException(94@[])
> at
> org.apache.phoenix.parse.PhoenixSQLParser.upsert_node(PhoenixSQLParser.java:4723)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:775)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:500)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108)
> ... 18 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407333#comment-15407333 ] Kalyan commented on PHOENIX-2084: - Hi Josh Mahonin, Current solution supports "json paths" like "f1.f2.f3", "f1.f2[*].f3" to access inner data from json object & json arrays. This is easy to define their "json paths" in simple property jackson is not providing this feature ... please refer this link .. https://github.com/jayway/JsonPath > Support loading json data using apache phoenix flume plugin > --- > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda >Assignee: venakt rao sirikonda > Fix For: 4.9.0 > > Attachments: phoenix_json.patch > > > We should support loading JSON data through our flume plugin as flume as > support for this data shape. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15407337#comment-15407337 ] Kalyan commented on PHOENIX-3135: - yes ... i agree with that, we can use it. > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Priority: Minor > Fix For: 4.9.0 > > Attachments: phoenix_csv.patch > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2290) Spark Phoenix cannot recognize Phoenix view fields
[
https://issues.apache.org/jira/browse/PHOENIX-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413260#comment-15413260
]
Kalyan commented on PHOENIX-2290:
-
this patch is working for above bug also:
https://issues.apache.org/jira/secure/attachment/12821623/phoenix_spark.patch
please review this
> Spark Phoenix cannot recognize Phoenix view fields
> --
>
> Key: PHOENIX-2290
> URL: https://issues.apache.org/jira/browse/PHOENIX-2290
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.5.1
>Reporter: Fengdong Yu
>Assignee: Josh Mahonin
> Labels: spark
>
> I created base table in base shell:
> {code}
> create 'test_table', {NAME => 'cf1', VERSIONS => 1}
> put 'test_table', 'row_key_1', 'cf1:col_1', '200'
> {code}
> This is a very simple table. then create phoenix view in Phoenix shell.
> {code}
> create view "test_table" (pk varchar primary key, "cf1"."col_1" varchar)
> {code}
> then do following in Spark shell:
> {code}
> val df = sqlContext.load("org.apache.phoenix.spark", Map("table" ->
> "\"test_table\"", "zkUrl" -> "localhost:2181"))
> df.registerTempTable("temp")
> {code}
> {code}
> scala> df.printSchema
> root
> |-- PK: string (nullable = true)
> |-- col_1: string (nullable = true)
> {code}
> sqlContext.sql("select * from temp") --> {color:red} This does
> work{color}
> then:
> {code}
> sqlContext.sql("select * from temp where col_1='200' ")
> {code}
> {code}
> java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=col_1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
> at
> org.apache.phoenix.spark.PhoenixRDD.getPartitions(PhoenixRDD.scala:47)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
> at scala.Option.getOrElse(Option.scala:120)
> {code}
> {color:red}
> I also tried:
> {code}
> sqlContext.sql("select * from temp where \"col_1\"='200' ") --> EMPTY
> result, no exception
> {code}
> {code}
> sqlContext.sql("select * from temp where \"cf1\".\"col_1\"='200' ") -->
> exception, cannot recognize SQL
> {code}
> {color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414873#comment-15414873
]
Kalyan commented on PHOENIX-2336:
-
problem with spark sql parser .. if expression contains "double quotes" ..
there are sending as usual to next level, with out "double quotes" there are
parsing using sql parser sending to next level.
i given a solution to handle in phoenix level to work. This patch is working
for above bug also:
https://issues.apache.org/jira/secure/attachment/12821623/phoenix_spark.patch
please review this
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414873#comment-15414873
]
Kalyan edited comment on PHOENIX-2336 at 8/10/16 8:00 AM:
--
problem with spark sql parser .. if expression contains "double quotes" ..
there are sending as usual to next level, with out "double quotes" there are
parsing using sql parser sending to next level.
i given a solution to handle in phoenix level to work. This patch is working
for above bug also:
https://issues.apache.org/jira/secure/attachment/12821623/phoenix_spark.patch
please review this.
// not working
DataFrame df =
sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
df.filter("\"col1\" = '5.0'").show();
But we need to write the query like this only... as per limitation
df.filter("col1 = '5.0'").show();
df.filter("col1 > '4.0'").show();
was (Author: kalyanhadoop):
problem with spark sql parser .. if expression contains "double quotes" ..
there are sending as usual to next level, with out "double quotes" there are
parsing using sql parser sending to next level.
i given a solution to handle in phoenix level to work. This patch is working
for above bug also:
https://issues.apache.org/jira/secure/attachment/12821623/phoenix_spark.patch
please review this
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417025#comment-15417025
]
Kalyan commented on PHOENIX-2336:
-
Hi Josh Mahonin,
same patch is going to work for PHOENIX-2336, PHOENIX-2290 and PHOENIX-2547.
i added the unit tests with proper comment also ..
Comments:
Limitation: filter / where expressions are not allowed with "double quotes",
instead of that pass it as column expressions
Reason: if the expression contains "double quotes" then spark sql parser,
ignoring evaluating .. giving to next level to handle
Please review this patch
https://github.com/kalyanhadooptraining/phoenix/commit/81df0c698ba4155a8f73ffe0ad657e9a5640d811
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kalyan updated PHOENIX-2336:
Attachment: PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_code_changes.patch
PHOENIX-2336, PHOENIX-2290 and PHOENIX-2547 code changes
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
> Attachments:
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_code_changes.patch,
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_unit_tests.patch
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kalyan updated PHOENIX-2336:
Attachment: PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_unit_tests.patch
PHOENIX-2336, PHOENIX-2290 and PHOENIX-2547 unit test cases
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
> Attachments:
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_code_changes.patch,
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_unit_tests.patch
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419065#comment-15419065
]
Kalyan commented on PHOENIX-2336:
-
Hi Josh Mahonin , updated patch attached .. please review
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
> Attachments:
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_code_changes.patch,
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_unit_tests.patch
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419656#comment-15419656
]
Kalyan commented on PHOENIX-2336:
-
Please go with '(Kalyan Hadoop)'
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Josh Mahonin
> Labels: verify
> Fix For: 4.9.0
>
> Attachments:
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_code_changes.patch,
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_unit_tests.patch
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2336) Queries with small case column-names return empty result-set when working with Spark Datasource Plugin
[
https://issues.apache.org/jira/browse/PHOENIX-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421837#comment-15421837
]
Kalyan commented on PHOENIX-2336:
-
Thanks Josh Mahonin & James Taylor
> Queries with small case column-names return empty result-set when working
> with Spark Datasource Plugin
> ---
>
> Key: PHOENIX-2336
> URL: https://issues.apache.org/jira/browse/PHOENIX-2336
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.6.0
>Reporter: Suhas Nalapure
>Assignee: Kalyan
> Labels: verify
> Fix For: 4.9.0, 4.8.1
>
> Attachments:
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_code_changes.patch,
> PHOENIX-2336_PHOENIX-2290_PHOENIX-2547_unit_tests.patch
>
>
> Hi,
> The Spark DataFrame filter operation returns empty result-set when
> column-name is in the smaller case. Example below:
> DataFrame df =
> sqlContext.read().format("org.apache.phoenix.spark").options(params).load();
> df.filter("\"col1\" = '5.0'").show();
> Result:
> +---++---+---+---+---
> | ID|col1| c1| d2| d3| d4|
> +---++---+---+---+---+
> +---++---+---+---+---+
> Whereas the table actually has some rows matching the filter condition. And
> if double quotes are removed from around the column name i.e. df.filter("col1
> = '5.0'").show(); , a ColumnNotFoundException is thrown:
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.phoenix.schema.ColumnNotFoundException: ERROR 504 (42703):
> Undefined column. columnName=D1
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
> at
> org.apache.phoenix.mapreduce.PhoenixInputFormat.getSplits(PhoenixInputFormat.java:80)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
Kalyan created PHOENIX-3185: --- Summary: Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514) Key: PHOENIX-3185 URL: https://issues.apache.org/jira/browse/PHOENIX-3185 Project: Phoenix Issue Type: Bug Reporter: Kalyan Assignee: Kalyan create a table with duplicate columns. create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, c3 double, c1 bigint); The below exception getting .. no issue with exception. But the problem is phoenix is creating table with exception & later we are not able to delete the table also. We need to fix this Bug 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, c3 double, c1 bigint); Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514) org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:808) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
[ https://issues.apache.org/jira/browse/PHOENIX-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3185: Attachment: image3.png image2.png image1.png Attached the screenshots > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > - > > Key: PHOENIX-3185 > URL: https://issues.apache.org/jira/browse/PHOENIX-3185 > Project: Phoenix > Issue Type: Bug >Reporter: Kalyan >Assignee: Kalyan > Attachments: image1.png, image2.png, image3.png > > > create a table with duplicate columns. > create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, > c3 double, c1 bigint); > The below exception getting .. no issue with exception. But the problem is > phoenix is creating table with exception & later we are not able to delete > the table also. We need to fix this Bug > 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary > key, c1 varchar, c2 integer, c3 double, c1 bigint); > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A > duplicate column name was detected in the object definition or ALTER TABLE > statement. columnName=TEST_TABLE.C1 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) > at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) > at > org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) > at sqlline.SqlLine.begin(SqlLine.java:681) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
[ https://issues.apache.org/jira/browse/PHOENIX-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422621#comment-15422621 ] Kalyan commented on PHOENIX-3185: - I just provided the solution to fix the bug please verify the solution patch: https://github.com/kalyanhadooptraining/phoenix/commit/80020c9f11f33e332fd8b01bf3630459e3af9851 any suggestions .. please update me. > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > - > > Key: PHOENIX-3185 > URL: https://issues.apache.org/jira/browse/PHOENIX-3185 > Project: Phoenix > Issue Type: Bug >Reporter: Kalyan >Assignee: Kalyan > Attachments: image1.png, image2.png, image3.png > > > create a table with duplicate columns. > create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, > c3 double, c1 bigint); > The below exception getting .. no issue with exception. But the problem is > phoenix is creating table with exception & later we are not able to delete > the table also. We need to fix this Bug > 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary > key, c1 varchar, c2 integer, c3 double, c1 bigint); > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A > duplicate column name was detected in the object definition or ALTER TABLE > statement. columnName=TEST_TABLE.C1 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) > at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) > at > org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) > at sqlline.SqlLine.begin(SqlLine.java:681) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
[ https://issues.apache.org/jira/browse/PHOENIX-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3185: Attachment: phoenix_duplicate_column_check.patch Please review the patch attached with unit test cases > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > - > > Key: PHOENIX-3185 > URL: https://issues.apache.org/jira/browse/PHOENIX-3185 > Project: Phoenix > Issue Type: Bug >Reporter: Kalyan >Assignee: Kalyan > Attachments: image1.png, image2.png, image3.png, > phoenix_duplicate_column_check.patch > > > create a table with duplicate columns. > create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, > c3 double, c1 bigint); > The below exception getting .. no issue with exception. But the problem is > phoenix is creating table with exception & later we are not able to delete > the table also. We need to fix this Bug > 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary > key, c1 varchar, c2 integer, c3 double, c1 bigint); > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A > duplicate column name was detected in the object definition or ALTER TABLE > statement. columnName=TEST_TABLE.C1 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) > at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) > at > org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) > at sqlline.SqlLine.begin(SqlLine.java:681) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan reassigned PHOENIX-2084: --- Assignee: Kalyan (was: venakt rao sirikonda) > Support loading json data using apache phoenix flume plugin > --- > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda >Assignee: Kalyan > Fix For: 4.9.0 > > Attachments: phoenix_json.patch > > > We should support loading JSON data through our flume plugin as flume as > support for this data shape. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan reassigned PHOENIX-3135: --- Assignee: Kalyan > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Fix For: 4.9.0 > > Attachments: phoenix_csv.patch > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-3074) Phoenix Bulkload fails when Table name contains '-' (hyphen)
[
https://issues.apache.org/jira/browse/PHOENIX-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kalyan reassigned PHOENIX-3074:
---
Assignee: Kalyan
> Phoenix Bulkload fails when Table name contains '-' (hyphen)
>
>
> Key: PHOENIX-3074
> URL: https://issues.apache.org/jira/browse/PHOENIX-3074
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.7.0
>Reporter: Dharmesh
>Assignee: Kalyan
> Labels: easyfix, features
> Fix For: 4.7.0
>
> Attachments: phoenix_hyphen.patch
>
>
> Bulk load fails from csv to Phoenix table when Phoenix table contains ‘-‘
> (hyphen) in table name. [Phoenix 4.7.0]
> Below is the steps to reproduce the issue.
> Step-1: create table with '-' hyphen. Use double quote around table name.
> CREATE TABLE "PHOENIX-TEST" (ID INTEGER PRIMARY KEY, NAME VARCHAR);
> Step-2:
> sudo -u hadoop
> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-protocol.jar:/usr/lib/hbase/conf/
> hadoop jar /usr/lib/phoenix/phoenix-client.jar
> org.apache.phoenix.mapreduce.CsvBulkLoadTool -Dfs.permissions.umask-mode=000
> -t "PHOENIX-TEST" --input "/user/test/PHOENIX-TEST.csv" -d $'\t'
> Below is the stack trace for the error. Ultimately it removes any double or
> single quotes around table name from the bulk load command.
> 2016-07-14 10:20:47,153 WARN [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> already initialized!
> 2016-07-14 10:20:48,703 INFO [main] org.apache.phoenix.util.UpsertExecutor:
> Upserting SQL data with UPSERT INTO PHOENIX-TEST ("ID", "0"."NAME") VALUES
> (?, ?)
> 2016-07-14 10:20:48,704 INFO [main] org.apache.hadoop.mapred.MapTask:
> Starting flush of map output
> 2016-07-14 10:20:48,711 INFO [main] org.apache.hadoop.io.compress.CodecPool:
> Got brand-new compressor [.snappy]
> 2016-07-14 10:20:48,718 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00):
> Syntax error. Encountered "-" at line 1, column 17.
> at
> org.apache.phoenix.util.UpsertExecutor.createStatement(UpsertExecutor.java:83)
> at org.apache.phoenix.util.UpsertExecutor.(UpsertExecutor.java:94)
> at
> org.apache.phoenix.util.csv.CsvUpsertExecutor.(CsvUpsertExecutor.java:63)
> at
> org.apache.phoenix.mapreduce.CsvToKeyValueMapper.buildUpsertExecutor(CsvToKeyValueMapper.java:85)
> at
> org.apache.phoenix.mapreduce.FormatToBytesWritableMapper.setup(FormatToBytesWritableMapper.java:142)
> at
> org.apache.phoenix.mapreduce.CsvToKeyValueMapper.setup(CsvToKeyValueMapper.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.phoenix.exception.PhoenixParserException: ERROR 601
> (42P00): Syntax error. Encountered "-" at line 1, column 17.
> at
> org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1185)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1268)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.(PhoenixPreparedStatement.java:94)
> at
> org.apache.phoenix.jdbc.PhoenixConnection.prepareStatement(PhoenixConnection.java:715)
> at
> org.apache.phoenix.util.UpsertExecutor.createStatement(UpsertExecutor.java:81)
> ... 13 more
> Caused by: NoViableAltException(94@[])
> at
> org.apache.phoenix.parse.PhoenixSQLParser.upsert_node(PhoenixSQLParser.java:4723)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:775)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:500)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108)
> ... 18 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (PHOENIX-2938) HFile support for SparkSQL DataFrame saves
[ https://issues.apache.org/jira/browse/PHOENIX-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan reassigned PHOENIX-2938: --- Assignee: Kalyan > HFile support for SparkSQL DataFrame saves > -- > > Key: PHOENIX-2938 > URL: https://issues.apache.org/jira/browse/PHOENIX-2938 > Project: Phoenix > Issue Type: Improvement >Reporter: Chris Tarnas >Assignee: Kalyan >Priority: Minor > > Currently when saving a DataFrame in Spark it is persisted as upserts. Having > an option to do saves natively via HFiles, as the MapReduce loader does, > would be a great performance improvement for large bulk loads. The current > work around to reduce the load on the regionservers would be to save to csv > from Spark then load via the MapReduce loader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2938) HFile support for SparkSQL DataFrame saves
[ https://issues.apache.org/jira/browse/PHOENIX-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422971#comment-15422971 ] Kalyan commented on PHOENIX-2938: - Converting HFile into SparkSQL DataFrame. Adding the existing base code to github https://github.com/kalyanhadooptraining/phoenix/commit/ce5869e3ae9036a72e123ff2e319ba0a1b59e922 TODO: 1. code cleanup 2. comments need to be update 3. unit test cases are required 4. final review on code any suggestions are allowed .. > HFile support for SparkSQL DataFrame saves > -- > > Key: PHOENIX-2938 > URL: https://issues.apache.org/jira/browse/PHOENIX-2938 > Project: Phoenix > Issue Type: Improvement >Reporter: Chris Tarnas >Assignee: Kalyan >Priority: Minor > > Currently when saving a DataFrame in Spark it is persisted as upserts. Having > an option to do saves natively via HFiles, as the MapReduce loader does, > would be a great performance improvement for large bulk loads. The current > work around to reduce the load on the regionservers would be to save to csv > from Spark then load via the MapReduce loader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
[ https://issues.apache.org/jira/browse/PHOENIX-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424782#comment-15424782 ] Kalyan commented on PHOENIX-3185: - Hi James Taylor, i think .. i need to follow the contributor guidelines for code format. my intention is to verify the patch only .. i feel it is a major bug. > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > - > > Key: PHOENIX-3185 > URL: https://issues.apache.org/jira/browse/PHOENIX-3185 > Project: Phoenix > Issue Type: Bug >Reporter: Kalyan >Assignee: Kalyan > Attachments: image1.png, image2.png, image3.png, > phoenix_duplicate_column_check.patch > > > create a table with duplicate columns. > create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, > c3 double, c1 bigint); > The below exception getting .. no issue with exception. But the problem is > phoenix is creating table with exception & later we are not able to delete > the table also. We need to fix this Bug > 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary > key, c1 varchar, c2 integer, c3 double, c1 bigint); > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A > duplicate column name was detected in the object definition or ALTER TABLE > statement. columnName=TEST_TABLE.C1 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) > at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) > at > org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) > at sqlline.SqlLine.begin(SqlLine.java:681) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-3192) phoenix-spark dataframe issue with combination of "column family + column name"
Kalyan created PHOENIX-3192:
---
Summary: phoenix-spark dataframe issue with combination of "column
family + column name"
Key: PHOENIX-3192
URL: https://issues.apache.org/jira/browse/PHOENIX-3192
Project: Phoenix
Issue Type: Bug
Reporter: Kalyan
Assignee: Kalyan
1. create table with different column families with same column name
create table tbl_1 (rowkey varchar primary key, cf1.c1 varchar, cf1.c2 integer,
cf2.c1 double, cf2.c2 boolean, cf3.c1 bigint);
2. insert sample data into table
3. create dataframe using phoenix table with different column names
val df1 = sqlContext.phoenixTableAsDataFrame("tbl_1", Array("CF1.C1",
"CF2.C2"), conf = configuration )
df1.show // this will work
4. create dataframe using phoenix table with same column names
val df2 = sqlContext.phoenixTableAsDataFrame("tbl_1", Array("CF1.C1",
"CF2.C1"), conf = configuration )
df2.show// this will fail
5. reason currently we are not handled the dataframe solution fully (column
family + column name).
only works with (column name)
Exception:
scala> val df2 = sqlContext.phoenixTableAsDataFrame("tbl_1", Array("CF1.C1",
"CF2.C1"), conf = configuration )
df2: org.apache.spark.sql.DataFrame = [C1: string, C1: double]
scala> df2.show
16/08/17 22:16:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
scala.MatchError: 1.5 (of class java.lang.Double)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:295)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:294)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:260)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401)
at
org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
at
org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
16/08/17 22:16:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
localhost): scala.MatchError: 1.5 (of class java.lang.Double)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:295)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:294)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTy
[jira] [Commented] (PHOENIX-2938) HFile support for SparkSQL DataFrame saves
[ https://issues.apache.org/jira/browse/PHOENIX-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424898#comment-15424898 ] Kalyan commented on PHOENIX-2938: - Thanks Josh Mahonin for your suggestions. i will work on these changes .. after the below bug fix. PHOENIX-3192 : phoenix-spark dataframe issue with combination of "column family + column name" otherwise i need to refactor again. > HFile support for SparkSQL DataFrame saves > -- > > Key: PHOENIX-2938 > URL: https://issues.apache.org/jira/browse/PHOENIX-2938 > Project: Phoenix > Issue Type: Improvement >Reporter: Chris Tarnas >Assignee: Kalyan >Priority: Minor > > Currently when saving a DataFrame in Spark it is persisted as upserts. Having > an option to do saves natively via HFiles, as the MapReduce loader does, > would be a great performance improvement for large bulk loads. The current > work around to reduce the load on the regionservers would be to save to csv > from Spark then load via the MapReduce loader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3192) phoenix-spark dataframe issue with combination of "column family + column name"
[
https://issues.apache.org/jira/browse/PHOENIX-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424914#comment-15424914
]
Kalyan commented on PHOENIX-3192:
-
I will be working on this bug .. planning to change the DataFrame creation with
schema mapping provided by user.
Otherwise we are not able to work with sql queries using sqlContext.sql()
operation.
Please share any suggestions to work with DataFrame fully (column family +
column name)
> phoenix-spark dataframe issue with combination of "column family + column
> name"
> ---
>
> Key: PHOENIX-3192
> URL: https://issues.apache.org/jira/browse/PHOENIX-3192
> Project: Phoenix
> Issue Type: Bug
>Reporter: Kalyan
>Assignee: Kalyan
>
> 1. create table with different column families with same column name
> create table tbl_1 (rowkey varchar primary key, cf1.c1 varchar, cf1.c2
> integer, cf2.c1 double, cf2.c2 boolean, cf3.c1 bigint);
> 2. insert sample data into table
> 3. create dataframe using phoenix table with different column names
> val df1 = sqlContext.phoenixTableAsDataFrame("tbl_1", Array("CF1.C1",
> "CF2.C2"), conf = configuration )
> df1.show // this will work
> 4. create dataframe using phoenix table with same column names
> val df2 = sqlContext.phoenixTableAsDataFrame("tbl_1", Array("CF1.C1",
> "CF2.C1"), conf = configuration )
> df2.show// this will fail
> 5. reason currently we are not handled the dataframe solution fully (column
> family + column name).
> only works with (column name)
> Exception:
> scala> val df2 = sqlContext.phoenixTableAsDataFrame("tbl_1", Array("CF1.C1",
> "CF2.C1"), conf = configuration )
> df2: org.apache.spark.sql.DataFrame = [C1: string, C1: double]
> scala> df2.show
> 16/08/17 22:16:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> scala.MatchError: 1.5 (of class java.lang.Double)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:295)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:294)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:260)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401)
> at
> org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
> at
> org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:492)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurre
[jira] [Commented] (PHOENIX-3185) Error: ERROR 514 (42892): A duplicate column name was detected in the object definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 (state=42892,code=514)
[ https://issues.apache.org/jira/browse/PHOENIX-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425944#comment-15425944 ] Kalyan commented on PHOENIX-3185: - Thanks William Yang , I am not aware of this issue then please make this issue as duplicate. James Taylor, please ignore my previous patch. sorry about my mistake. > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > - > > Key: PHOENIX-3185 > URL: https://issues.apache.org/jira/browse/PHOENIX-3185 > Project: Phoenix > Issue Type: Bug >Reporter: Kalyan >Assignee: Kalyan > Fix For: 4.8.1 > > Attachments: image1.png, image2.png, image3.png, > phoenix_duplicate_column_check.patch > > > create a table with duplicate columns. > create table test_table (rowkey varchar primary key, c1 varchar, c2 integer, > c3 double, c1 bigint); > The below exception getting .. no issue with exception. But the problem is > phoenix is creating table with exception & later we are not able to delete > the table also. We need to fix this Bug > 0: jdbc:phoenix:localhost> create table test_table (rowkey varchar primary > key, c1 varchar, c2 integer, c3 double, c1 bigint); > Error: ERROR 514 (42892): A duplicate column name was detected in the object > definition or ALTER TABLE statement. columnName=TEST_TABLE.C1 > (state=42892,code=514) > org.apache.phoenix.schema.ColumnAlreadyExistsException: ERROR 514 (42892): A > duplicate column name was detected in the object definition or ALTER TABLE > statement. columnName=TEST_TABLE.C1 > at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:415) > at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:315) > at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:288) > at > org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2146) > at > org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828) > at > org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1345) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) > at sqlline.SqlLine.begin(SqlLine.java:681) > at sqlline.SqlLine.start(SqlLine.java:398) > at sqlline.SqlLine.main(SqlLine.java:292) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-3214) Kafka Phoenix Consumer
Kalyan created PHOENIX-3214: --- Summary: Kafka Phoenix Consumer Key: PHOENIX-3214 URL: https://issues.apache.org/jira/browse/PHOENIX-3214 Project: Phoenix Issue Type: New Feature Reporter: Kalyan Assignee: Kalyan Providing a new feature to Phoenix. Directly ingest Kafka messages to Phoenix. Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448913#comment-15448913 ] Kalyan commented on PHOENIX-3214: - Providing a new feature to Phoenix. Directly ingest Kafka messages to Phoenix. Similar to flume phoenix integration. Implementation is done using phoenix-flume solution for inserting the kafka messages. Please review the patch: https://github.com/kalyanhadooptraining/phoenix/commit/b35a7fb8ffb1fd7745a0467a4cea3b761459e463 > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15521599#comment-15521599 ] Kalyan commented on PHOENIX-2084: - This solution is completed .. patch is also attached, you can use also. May anyone can verify & committing is left now. > Support loading json data using apache phoenix flume plugin > --- > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda >Assignee: Kalyan > Fix For: 4.9.0 > > Attachments: phoenix_json.patch > > > We should support loading JSON data through our flume plugin as flume as > support for this data shape. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-2084) Support loading json data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15521599#comment-15521599 ] Kalyan edited comment on PHOENIX-2084 at 9/25/16 11:31 PM: --- This solution is completed .. patch is also attached, you can use also. May be anyone can verify & committing is left now. was (Author: kalyanhadoop): This solution is completed .. patch is also attached, you can use also. May anyone can verify & committing is left now. > Support loading json data using apache phoenix flume plugin > --- > > Key: PHOENIX-2084 > URL: https://issues.apache.org/jira/browse/PHOENIX-2084 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: venakt rao sirikonda >Assignee: Kalyan > Fix For: 4.9.0 > > Attachments: phoenix_json.patch > > > We should support loading JSON data through our flume plugin as flume as > support for this data shape. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15521602#comment-15521602 ] Kalyan commented on PHOENIX-3135: - This solution is completed .. patch is also attached, you can use also. May be anyone can verify & committing is left now. > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Fix For: 4.9.0 > > Attachments: phoenix_csv.patch > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-3311) phoenix-spark("4.8.0-HBase-1.2") is not compatible with spark 2.0
[
https://issues.apache.org/jira/browse/PHOENIX-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kalyan reassigned PHOENIX-3311:
---
Assignee: Kalyan
> phoenix-spark("4.8.0-HBase-1.2") is not compatible with spark 2.0
> -
>
> Key: PHOENIX-3311
> URL: https://issues.apache.org/jira/browse/PHOENIX-3311
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Kui Xiang
>Assignee: Kalyan
>Priority: Critical
> Labels: phoenix, spark2.0.0
>
> sbt:
> libraryDependencies += "org.apache.phoenix" % "phoenix-spark" %
> "4.8.0-HBase-1.2"
> scala:
> import org.apache.phoenix.spark._
> will compile fail with errors like
> [error] missing or invalid dependency detected while loading class file
> 'ProductRDDFunctions.class'.
> [error] Could not access type Logging in package org.apache.spark,
> [error] because it (or its dependencies) are missing. Check your build
> definition for
> [error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath`
> to see the problematic classpath.)
> [error] A full rebuild may help if 'ProductRDDFunctions.class' was compiled
> against an incompatible version of org.apache.spark.
> [error] one error found
> [debug] Compilation failed (CompilerInterface)
> [error] (compile:compileIncremental) Compilation failed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3311) phoenix-spark("4.8.0-HBase-1.2") is not compatible with spark 2.0
[
https://issues.apache.org/jira/browse/PHOENIX-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15694941#comment-15694941
]
Kalyan commented on PHOENIX-3311:
-
please verify the patch to work with spark-1.x and spark-2.x
https://github.com/kalyanhadooptraining/phoenix/commit/98cf1b408358c0f9687b1aadf91ede64fdc0a05d
> phoenix-spark("4.8.0-HBase-1.2") is not compatible with spark 2.0
> -
>
> Key: PHOENIX-3311
> URL: https://issues.apache.org/jira/browse/PHOENIX-3311
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Kui Xiang
>Assignee: Kalyan
>Priority: Critical
> Labels: phoenix, spark2.0.0
>
> sbt:
> libraryDependencies += "org.apache.phoenix" % "phoenix-spark" %
> "4.8.0-HBase-1.2"
> scala:
> import org.apache.phoenix.spark._
> will compile fail with errors like
> [error] missing or invalid dependency detected while loading class file
> 'ProductRDDFunctions.class'.
> [error] Could not access type Logging in package org.apache.spark,
> [error] because it (or its dependencies) are missing. Check your build
> definition for
> [error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath`
> to see the problematic classpath.)
> [error] A full rebuild may help if 'ProductRDDFunctions.class' was compiled
> against an incompatible version of org.apache.spark.
> [error] one error found
> [debug] Compilation failed (CompilerInterface)
> [error] (compile:compileIncremental) Compilation failed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3311) phoenix-spark("4.8.0-HBase-1.2") is not compatible with spark 2.0
[
https://issues.apache.org/jira/browse/PHOENIX-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15696880#comment-15696880
]
Kalyan commented on PHOENIX-3311:
-
Thanks Josh Mahonin , Both solutions are duplicate only ... we can ignore this
solution.
I tried with spark-1.6 with scala-2.10 and spark-2.0 with scala-2.11 ... code
working.
we need to prepare 2 jars with old and new version support.
> phoenix-spark("4.8.0-HBase-1.2") is not compatible with spark 2.0
> -
>
> Key: PHOENIX-3311
> URL: https://issues.apache.org/jira/browse/PHOENIX-3311
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.8.0
>Reporter: Kui Xiang
>Assignee: Kalyan
>Priority: Critical
> Labels: phoenix, spark2.0.0
>
> sbt:
> libraryDependencies += "org.apache.phoenix" % "phoenix-spark" %
> "4.8.0-HBase-1.2"
> scala:
> import org.apache.phoenix.spark._
> will compile fail with errors like
> [error] missing or invalid dependency detected while loading class file
> 'ProductRDDFunctions.class'.
> [error] Could not access type Logging in package org.apache.spark,
> [error] because it (or its dependencies) are missing. Check your build
> definition for
> [error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath`
> to see the problematic classpath.)
> [error] A full rebuild may help if 'ProductRDDFunctions.class' was compiled
> against an incompatible version of org.apache.spark.
> [error] one error found
> [debug] Compilation failed (CompilerInterface)
> [error] (compile:compileIncremental) Compilation failed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752882#comment-15752882 ] Kalyan commented on PHOENIX-3135: - Hi .. Can anyone look into this patch. Thanks > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Fix For: 4.10.0 > > Attachments: phoenix_csv.patch > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-3538) Regex Bulkload Tool
Kalyan created PHOENIX-3538: --- Summary: Regex Bulkload Tool Key: PHOENIX-3538 URL: https://issues.apache.org/jira/browse/PHOENIX-3538 Project: Phoenix Issue Type: New Feature Reporter: Kalyan Assignee: Kalyan Priority: Minor To work with complex data , we can regex to load directly. Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752899#comment-15752899 ] Kalyan commented on PHOENIX-3538: - I will be working on this tool > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan reassigned PHOENIX-3541: --- Assignee: Kalyan > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759080#comment-15759080 ] Kalyan commented on PHOENIX-3538: - Please verify the patch below https://bitbucket.org/kalyanhadooptraining/phoenix/commits/8feabadb725265e65199a68aa7ddf61c74dbaf3e > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3538: Attachment: PHOENIX-3538.patch providing the solution to regex bulk load tool > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3541: Attachment: PHOENIX-3541.patch Please review the patch ... > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > Attachments: PHOENIX-3541.patch > > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759731#comment-15759731 ] Kalyan edited comment on PHOENIX-3541 at 12/19/16 12:06 AM: Please review the patch ... https://bitbucket.org/kalyanhadooptraining/phoenix/commits/2f3fd851253bcfce288e350ac50ebed66dcf630e was (Author: kalyanhadoop): Please review the patch ... > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > Attachments: PHOENIX-3541.patch > > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759737#comment-15759737 ] Kalyan commented on PHOENIX-3214: - Can any one please look into the patch ... any suggestions ? > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15759792#comment-15759792 ] Kalyan commented on PHOENIX-3214: - we can reuse it ... the below line will take care of it. this.consumer = new KafkaConsumer<>(properties); https://github.com/kalyanhadooptraining/phoenix/commit/b35a7fb8ffb1fd7745a0467a4cea3b761459e463#diff-475a67c1e9b3f68ef5d95f4ce536bb04R121 > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15760418#comment-15760418 ] Kalyan commented on PHOENIX-3214: - Kevin Liew, thanks for review. Updated as per suggestion. https://github.com/kalyanhadooptraining/phoenix/commit/d6d8061f554e52ab692414affc115386a006a647 > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3214: Attachment: PHOENIX-3214.patch > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761866#comment-15761866 ] Kalyan commented on PHOENIX-3214: - Hi Josh Mahonin, Please review the patch. Thanks > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3214: Attachment: PHOENIX-3214-final.patch > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-final.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761938#comment-15761938 ] Kalyan commented on PHOENIX-3214: - Hi Josh Mahonin, Sorry i missed those files in previous patch. Updated final patch. Thanks > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-final.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763467#comment-15763467 ] Kalyan commented on PHOENIX-3541: - follow this link: http://phoenix.apache.org/building.html `mvn package` command will build > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > Attachments: PHOENIX-3541.patch > > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3538: Attachment: PHOENIX-3538-v1.patch > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-v1.patch, PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763645#comment-15763645 ] Kalyan commented on PHOENIX-3538: - PHOENIX-3485 : CSVBulkLoadToolIT failing consistently because of HBASE-17108 similar checkin in RegexBulkLoadToolIT.java > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-v1.patch, PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763645#comment-15763645 ] Kalyan edited comment on PHOENIX-3538 at 12/20/16 8:40 AM: --- PHOENIX-3485 : CSVBulkLoadToolIT failing consistently because of HBASE-17108 similar checkin in RegexBulkLoadToolIT.java https://bitbucket.org/kalyanhadooptraining/phoenix/commits/8aa487b72292b0dacbc539e7fad9ee8bc9aca9c9 was (Author: kalyanhadoop): PHOENIX-3485 : CSVBulkLoadToolIT failing consistently because of HBASE-17108 similar checkin in RegexBulkLoadToolIT.java > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-v1.patch, PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763759#comment-15763759 ] Kalyan commented on PHOENIX-3541: - execute like below it will work hadoop jar /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool --t "\"\"abc\"\"" --input /event_outb/example.csv > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > Attachments: PHOENIX-3541.patch > > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763759#comment-15763759 ] Kalyan edited comment on PHOENIX-3541 at 12/20/16 9:35 AM: --- execute like below it will work hadoop jar /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool --t "\"\"abc\"\"" --input /event_outb/example.csv or hadoop jar /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool --t \"\"abc\"\" --input /event_outb/example.csv was (Author: kalyanhadoop): execute like below it will work hadoop jar /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool --t "\"\"abc\"\"" --input /event_outb/example.csv > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > Attachments: PHOENIX-3541.patch > > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3538: Attachment: PHOENIX-3538-final.patch > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-final.patch, PHOENIX-3538-v1.patch, > PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765714#comment-15765714 ] Kalyan commented on PHOENIX-3538: - Hi James Taylor, updated final patch attached (PHOENIX-3538-final.patch). Please review this `PHOENIX-3538-final.patch`. Thanks > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-final.patch, PHOENIX-3538-v1.patch, > PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3541) Bulk Data Loading - Can't use table name by small letter
[ https://issues.apache.org/jira/browse/PHOENIX-3541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765995#comment-15765995 ] Kalyan commented on PHOENIX-3541: - it is compatibility issue with versions of phoenix ... it works the patch i given and also test case also added for this .. you can verify > Bulk Data Loading - Can't use table name by small letter > - > > Key: PHOENIX-3541 > URL: https://issues.apache.org/jira/browse/PHOENIX-3541 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.8.0 >Reporter: Beomjun Kim >Assignee: Kalyan >Priority: Critical > Attachments: PHOENIX-3541.patch > > > i have existing Phoenix table abc > i wanna Bulk Data Loading via MapReduce > And then used the following command to load the csv file > hadoop jar > /root/Phoenix/apache-phoenix-4.8.0-HBase-0.98-bin/phoenix-4.8.0-HBase-0.98-client.jar > org.apache.phoenix.mapreduce.CsvBulkLoadTool --t abc --input /example.csv > but, it does not seem to find the table abc > Exception in thread "main" java.lang.IllegalArgumentException: Table ABC not > found > i try change command table name --t 'abc' and --t "abc" > but it doesn't work > how can i use table name small letter ?? > And also, i found same case > http://apache-phoenix-user-list.1124778.n5.nabble.com/Load-into-Phoenix-table-via-CsvBulkLoadTool-cannot-find-table-and-fails-td2792.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3333) Support Spark 2.0
[
https://issues.apache.org/jira/browse/PHOENIX-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766002#comment-15766002
]
Kalyan commented on PHOENIX-:
-
the patch is working fine ... from my end no issues
> Support Spark 2.0
> -
>
> Key: PHOENIX-
> URL: https://issues.apache.org/jira/browse/PHOENIX-
> Project: Phoenix
> Issue Type: Improvement
>Affects Versions: 4.9.1
> Environment: spark 2.0 ,phoenix 4.8.0 , os is centos 6.7 ,hadoop is
> hdp 2.5
>Reporter: dalin qin
> Fix For: 4.10.0
>
> Attachments: PHOENIX--interim.patch, PHOENIX-.patch
>
>
> spark version is 2.0.0.2.5.0.0-1245
> As mentioned by Josh , I believe spark 2.0 changed their api so that failed
> phoenix. Please come up with update version to adapt spark's change.
> In [1]: df = sqlContext.read \
>...: .format("org.apache.phoenix.spark") \
>...: .option("table", "TABLE1") \
>...: .option("zkUrl", "namenode:2181:/hbase-unsecure") \
>...: .load()
> ---
> Py4JJavaError Traceback (most recent call last)
> in ()
> > 1 df = sqlContext.read .format("org.apache.phoenix.spark")
> .option("table", "TABLE1") .option("zkUrl",
> "namenode:2181:/hbase-unsecure") .load()
> /usr/hdp/2.5.0.0-1245/spark2/python/pyspark/sql/readwriter.pyc in load(self,
> path, format, schema, **options)
> 151 return
> self._df(self._jreader.load(self._spark._sc._jvm.PythonUtils.toSeq(path)))
> 152 else:
> --> 153 return self._df(self._jreader.load())
> 154
> 155 @since(1.4)
> /usr/hdp/2.5.0.0-1245/spark2/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py
> in __call__(self, *args)
> 931 answer = self.gateway_client.send_command(command)
> 932 return_value = get_return_value(
> --> 933 answer, self.gateway_client, self.target_id, self.name)
> 934
> 935 for temp_arg in temp_args:
> /usr/hdp/2.5.0.0-1245/spark2/python/pyspark/sql/utils.pyc in deco(*a, **kw)
> 61 def deco(*a, **kw):
> 62 try:
> ---> 63 return f(*a, **kw)
> 64 except py4j.protocol.Py4JJavaError as e:
> 65 s = e.java_exception.toString()
> /usr/hdp/2.5.0.0-1245/spark2/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py
> in get_return_value(answer, gateway_client, target_id, name)
> 310 raise Py4JJavaError(
> 311 "An error occurred while calling {0}{1}{2}.\n".
> --> 312 format(target_id, ".", name), value)
> 313 else:
> 314 raise Py4JError(
> Py4JJavaError: An error occurred while calling o43.load.
> : java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethod(Class.java:2128)
> at
> java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)
> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.(ObjectStreamClass.java:472)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> at
> org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:295)
> at
> org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
> at
> org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
> at org.apache.spark.SparkContext.clean(SparkContext.scala:2037)
> at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:366)
> at org.apache.spark.rdd.RDD$$anonfun$map$1
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767196#comment-15767196 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin] This integration internal uses `phoenix-flume plugin` only. I am converting streaming `Kakfa messages` as a `Flume events` then using total `phoenix-flume plugin` to insert into Phoenix Tables. I made as a separate `phoenix-kafka` module and provided the dependency as a `phoenix-flume` module. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-final.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3214: Attachment: PHOENIX-3214-updated.patch PHOENIX-3214-docs.patch Hi [~jmahonin] 1. added integration document for phoenix-kafka. 2. provided command line support to execute this tool. Attached files are : PHOENIX-3214-docs.patch & PHOENIX-3214-updated.patch. Ignore previous patches. Thanks for review > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15775627#comment-15775627 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin] can please review it .. it is completed with test cases and document. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15776569#comment-15776569 ] Kalyan commented on PHOENIX-3214: - Thanks [~giacomotaylor] , [~jmahonin] > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810215#comment-15810215 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin], can you please review it > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15810219#comment-15810219 ] Kalyan commented on PHOENIX-3538: - Hi [~jmahonin], [~ndimiduk], [~gabriel.reid], [~maghamraviki...@gmail.com] can any one please review it > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-final.patch, PHOENIX-3538-v1.patch, > PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3538: Attachment: PHOENIX-3538-codecleanup.patch Thanks, [~gabriel.reid] As per [~gabriel.reid] comments updated the code 1. removed the duplicate code 2. added javadocs attaching new patch. Can you please review once again [~gabriel.reid] > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-codecleanup.patch, > PHOENIX-3538-final.patch, PHOENIX-3538-v1.patch, PHOENIX-3538.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3214: Attachment: PHOENIX-3214-updated-1.patch Hi [~jmahonin] created new patch on master branch. can you please verify once i tested with "git apply PHOENIX-3214-updated-1.patch", it is worked for me. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3214: Attachment: PHOENIX-3214-updated-2.patch Hi [~jmahonin], [~giacomotaylor] Previous patch Test Case is failed due to dependency issues in pom.xml. Now updated pom.xml file and created new patch file "PHOENIX-3214-updated-2.patch". Can you please verify once .. thanks for your time. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854284#comment-15854284 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin], [~elserj], [~ankit.singhal], i am working on this. let me try to fix .. problem only in pom.xml file in parent. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854323#comment-15854323 ] Kalyan commented on PHOENIX-3214: - Hi [~elserj], Thanks for your support. I also verified your patches .. working fine. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3135) Support loading csv data using apache phoenix flume plugin
[ https://issues.apache.org/jira/browse/PHOENIX-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867272#comment-15867272 ] Kalyan commented on PHOENIX-3135: - Hi [~jmahonin], [~jamestaylor] , we can change this. Sorry about my mistake. Thanks > Support loading csv data using apache phoenix flume plugin > -- > > Key: PHOENIX-3135 > URL: https://issues.apache.org/jira/browse/PHOENIX-3135 > Project: Phoenix > Issue Type: New Feature > Environment: cloudera 5.4 >Reporter: Kalyan >Assignee: Josh Mahonin >Priority: Minor > Fix For: 4.10.0 > > Attachments: phoenix_csv.patch > > > To work with below sample data sets ... we need support loading csv data > using apache phoenix flume plugin. > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar, col4 integer > input: kalyan,10.5,abc,1 > input: "kalyan",10.5,"abc",1 > // sample data set 1 > schema: col1 varchar , col2 double, col3 varchar[], col4 integer[] > input: kalyan,10.5,"abc,pqr,xyz","1,2,3,4" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3658) Remove org.json:json dependency from flume module
[ https://issues.apache.org/jira/browse/PHOENIX-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867296#comment-15867296 ] Kalyan commented on PHOENIX-3658: - Hi [~jmahonin], we can go for com:tdunning:json, no issues from my side. Thanks > Remove org.json:json dependency from flume module > - > > Key: PHOENIX-3658 > URL: https://issues.apache.org/jira/browse/PHOENIX-3658 > Project: Phoenix > Issue Type: Task >Reporter: Josh Elser >Assignee: Josh Mahonin >Priority: Blocker > Attachments: PHOENIX-3658.patch > > > The phoenix-flume module depends on org.json:json which is now category-x. > We have a grace period until 2017/04/30 to resolve this one. > Need to replace it with something else. > https://www.apache.org/legal/resolved#json > https://lists.apache.org/thread.html/bb18f942ce7eb83c11438303c818b885810fb76385979490366720d5@%3Clegal-discuss.apache.org%3E -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kalyan updated PHOENIX-3538: Attachment: PHOENIX-3538.src.patch PHOENIX-3538.docs.patch Hi [~gabriel.reid], [~jamestaylor] Attaching the updated patches : PHOENIX-3538.src.patch and PHOENIX-3538.docs.patch As per [~gabriel.reid] comments updated the test cases and docs Below task is not done. I am not getting generically-name to these classes. `could you move the json-specific class that is being used for both the regex loader and json loader into a more generically-named package and class?` Thanks for review > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Attachments: PHOENIX-3538-codecleanup.patch, PHOENIX-3538.docs.patch, > PHOENIX-3538-final.patch, PHOENIX-3538.patch, PHOENIX-3538.src.patch, > PHOENIX-3538-v1.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3538) Regex Bulkload Tool
[ https://issues.apache.org/jira/browse/PHOENIX-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874186#comment-15874186 ] Kalyan commented on PHOENIX-3538: - Thanks [~gabriel.reid], [~jamestaylor] > Regex Bulkload Tool > --- > > Key: PHOENIX-3538 > URL: https://issues.apache.org/jira/browse/PHOENIX-3538 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan >Priority: Minor > Fix For: 4.10.0 > > Attachments: PHOENIX-3538-codecleanup.patch, PHOENIX-3538.docs.patch, > PHOENIX-3538-final.patch, PHOENIX-3538.patch, PHOENIX-3538.src.patch, > PHOENIX-3538-v1.patch > > > To work with complex data , we can regex to load directly. > Similar to JSON Bulkload Tool & CSV Bulkload Tool -- This message was sent by Atlassian JIRA (v6.3.15#6346)
