[jira] [Commented] (PHOENIX-1071) Provide integration for exposing Phoenix tables as Spark RDDs
[
https://issues.apache.org/jira/browse/PHOENIX-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396341#comment-14396341
]
Josh Mahonin commented on PHOENIX-1071:
---
Hmm, sorry about that. I suspect it's related to the 32-bit VM, as per:
http://stackoverflow.com/questions/4401396/could-not-reserve-enough-space-for-object-heap
I don't have an environment to reproduce the issue myself, but some tweaking of
'-XX:MaxHeapSize' and/or '-Xmx' would probably get things building again.
> Provide integration for exposing Phoenix tables as Spark RDDs
> -
>
> Key: PHOENIX-1071
> URL: https://issues.apache.org/jira/browse/PHOENIX-1071
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Andrew Purtell
>
> A core concept of Apache Spark is the resilient distributed dataset (RDD), a
> "fault-tolerant collection of elements that can be operated on in parallel".
> One can create a RDDs referencing a dataset in any external storage system
> offering a Hadoop InputFormat, like PhoenixInputFormat and
> PhoenixOutputFormat. There could be opportunities for additional interesting
> and deep integration.
> Add the ability to save RDDs back to Phoenix with a {{saveAsPhoenixTable}}
> action, implicitly creating necessary schema on demand.
> Add support for {{filter}} transformations that push predicates to the server.
> Add a new {{select}} transformation supporting a LINQ-like DSL, for example:
> {code}
> // Count the number of different coffee varieties offered by each
> // supplier from Guatemala
> phoenixTable("coffees")
> .select(c =>
> where(c.origin == "GT"))
> .countByKey()
> .foreach(r => println(r._1 + "=" + r._2))
> {code}
> Support conversions between Scala and Java types and Phoenix table data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-1815) Use Spark Data Source API in phoenix-spark module
Josh Mahonin created PHOENIX-1815: - Summary: Use Spark Data Source API in phoenix-spark module Key: PHOENIX-1815 URL: https://issues.apache.org/jira/browse/PHOENIX-1815 Project: Phoenix Issue Type: New Feature Reporter: Josh Mahonin Spark 1.3.0 introduces a new 'Data Source' API to standardize load and save methods for different types of data sources. The phoenix-spark module should implement the same API for use as a pluggable data store in Spark. ref: https://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1816) Add markdown page on Phoenix website describing how to use Spark integration
[ https://issues.apache.org/jira/browse/PHOENIX-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482156#comment-14482156 ] Josh Mahonin commented on PHOENIX-1816: --- Sure thing. I think I've got the good parts already covered in the existing README.md, but I can flush it out some more to follow the documentation style of the rest of the site. It's been a while since I've had to use svn. Do I need to get any commit privileges set, or should I attach a patch to this ticket? > Add markdown page on Phoenix website describing how to use Spark integration > > > Key: PHOENIX-1816 > URL: https://issues.apache.org/jira/browse/PHOENIX-1816 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > > Now that we have some nifty integration with Spark, we should add a markdown > page to our site (http://phoenix.apache.org/), with a link in the Using menu > that describes how to use it. > See http://phoenix.apache.org/building_website.html for how to modify the the > website (in lives in svn). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1818) Move cluster-required tests to src/it
[ https://issues.apache.org/jira/browse/PHOENIX-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482166#comment-14482166 ] Josh Mahonin commented on PHOENIX-1818: --- Makes sense, I'll try get them moved over this week. I think ideally I'd like to see what changes are needed first to get the existing tests working on Jenkins at builds.apache.org. I see there's been some changes made in master for the memory settings in the phoenix-spark/pom.xml, but they're still failing. I haven't had a chance yet to spin up an environment to reproduce them myself. > Move cluster-required tests to src/it > - > > Key: PHOENIX-1818 > URL: https://issues.apache.org/jira/browse/PHOENIX-1818 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > > Longer running unit tests should be placed under src/it and run when "mvn > verify" is executed. Short running unit tests can remain under src/test. See > phoenix-core for an example -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1816) Add markdown page on Phoenix website describing how to use Spark integration
[ https://issues.apache.org/jira/browse/PHOENIX-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1816: -- Attachment: README.md > Add markdown page on Phoenix website describing how to use Spark integration > > > Key: PHOENIX-1816 > URL: https://issues.apache.org/jira/browse/PHOENIX-1816 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > Attachments: README.md > > > Now that we have some nifty integration with Spark, we should add a markdown > page to our site (http://phoenix.apache.org/), with a link in the Using menu > that describes how to use it. > See http://phoenix.apache.org/building_website.html for how to modify the the > website (in lives in svn). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1816) Add markdown page on Phoenix website describing how to use Spark integration
[ https://issues.apache.org/jira/browse/PHOENIX-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483240#comment-14483240 ] Josh Mahonin commented on PHOENIX-1816: --- Added the same README.md as is changes in this PR: https://github.com/apache/phoenix/pull/64 I think it makes sense to have them in sync? Let me know if you need any modifications done, I tried to follow the same style as the other markdown files. > Add markdown page on Phoenix website describing how to use Spark integration > > > Key: PHOENIX-1816 > URL: https://issues.apache.org/jira/browse/PHOENIX-1816 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > Attachments: README.md > > > Now that we have some nifty integration with Spark, we should add a markdown > page to our site (http://phoenix.apache.org/), with a link in the Using menu > that describes how to use it. > See http://phoenix.apache.org/building_website.html for how to modify the the > website (in lives in svn). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1071) Provide integration for exposing Phoenix tables as Spark RDDs
[
https://issues.apache.org/jira/browse/PHOENIX-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484574#comment-14484574
]
Josh Mahonin commented on PHOENIX-1071:
---
It looks like the latest changes that moved the unit tests into the 'it'
directory end up actually disabling them, I think due to the filename not
ending in "*IT". Due to the memory / VM complications, this might be ideal for
now.
I have a local branch with the unit tests running during the integration-test
phase, and I'm presently working on having the tests extend the
BaseHBaseManagedTimeIT interface. My hope is that once that's in place, the
build issues and parallel execution limitions will be fixed along with it. If
all goes well I should have something ready tomorrow.
> Provide integration for exposing Phoenix tables as Spark RDDs
> -
>
> Key: PHOENIX-1071
> URL: https://issues.apache.org/jira/browse/PHOENIX-1071
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Andrew Purtell
>Assignee: Josh Mahonin
>
> A core concept of Apache Spark is the resilient distributed dataset (RDD), a
> "fault-tolerant collection of elements that can be operated on in parallel".
> One can create a RDDs referencing a dataset in any external storage system
> offering a Hadoop InputFormat, like PhoenixInputFormat and
> PhoenixOutputFormat. There could be opportunities for additional interesting
> and deep integration.
> Add the ability to save RDDs back to Phoenix with a {{saveAsPhoenixTable}}
> action, implicitly creating necessary schema on demand.
> Add support for {{filter}} transformations that push predicates to the server.
> Add a new {{select}} transformation supporting a LINQ-like DSL, for example:
> {code}
> // Count the number of different coffee varieties offered by each
> // supplier from Guatemala
> phoenixTable("coffees")
> .select(c =>
> where(c.origin == "GT"))
> .countByKey()
> .foreach(r => println(r._1 + "=" + r._2))
> {code}
> Support conversions between Scala and Java types and Phoenix table data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-1071) Provide integration for exposing Phoenix tables as Spark RDDs
[
https://issues.apache.org/jira/browse/PHOENIX-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485440#comment-14485440
]
Josh Mahonin commented on PHOENIX-1071:
---
Works for me on both master and 4.x-HBase-0.98
PhoenixSparkIT also runs from IntelliJ using the 'ScalaTest' configuration with
the following VM parameters:
-Xmx1536m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=512m
> Provide integration for exposing Phoenix tables as Spark RDDs
> -
>
> Key: PHOENIX-1071
> URL: https://issues.apache.org/jira/browse/PHOENIX-1071
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Andrew Purtell
>Assignee: Josh Mahonin
>
> A core concept of Apache Spark is the resilient distributed dataset (RDD), a
> "fault-tolerant collection of elements that can be operated on in parallel".
> One can create a RDDs referencing a dataset in any external storage system
> offering a Hadoop InputFormat, like PhoenixInputFormat and
> PhoenixOutputFormat. There could be opportunities for additional interesting
> and deep integration.
> Add the ability to save RDDs back to Phoenix with a {{saveAsPhoenixTable}}
> action, implicitly creating necessary schema on demand.
> Add support for {{filter}} transformations that push predicates to the server.
> Add a new {{select}} transformation supporting a LINQ-like DSL, for example:
> {code}
> // Count the number of different coffee varieties offered by each
> // supplier from Guatemala
> phoenixTable("coffees")
> .select(c =>
> where(c.origin == "GT"))
> .countByKey()
> .foreach(r => println(r._1 + "=" + r._2))
> {code}
> Support conversions between Scala and Java types and Phoenix table data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-1815) Use Spark Data Source API in phoenix-spark module
[ https://issues.apache.org/jira/browse/PHOENIX-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1815: -- Attachment: 4x-098_1815.patch master_1815.patch PHOENIX-1815 patches for both master and 4.x-HBase-0.98 branch > Use Spark Data Source API in phoenix-spark module > - > > Key: PHOENIX-1815 > URL: https://issues.apache.org/jira/browse/PHOENIX-1815 > Project: Phoenix > Issue Type: New Feature >Reporter: Josh Mahonin > Attachments: 4x-098_1815.patch, master_1815.patch > > > Spark 1.3.0 introduces a new 'Data Source' API to standardize load and save > methods for different types of data sources. > The phoenix-spark module should implement the same API for use as a pluggable > data store in Spark. > ref: > https://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources > > https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1877) Spark end2end tests are breaking the build
[ https://issues.apache.org/jira/browse/PHOENIX-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1877: -- Attachment: PHOENIX-1877.patch > Spark end2end tests are breaking the build > -- > > Key: PHOENIX-1877 > URL: https://issues.apache.org/jira/browse/PHOENIX-1877 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-1877.patch > > > See below for the error. Maybe we should disable these tests until this can > be sorted out? > [31m*** RUN ABORTED *** [0m > [31m java.util.concurrent.ExecutionException: > org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): > Syntax error. Encountered "" at line 1, column 62. [0m > [31m at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > [0m > [31m at java.util.concurrent.FutureTask.get(FutureTask.java:111) [0m > [31m at > org.scalatest.tools.ConcurrentDistributor.waitUntilDone(ConcurrentDistributor.scala:52) > [0m > [31m at > org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2549) [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044) > [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043) > [0m > [31m at > org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722) > [0m > [31m at > org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043) > [0m > [31m at org.scalatest.tools.Runner$.main(Runner.scala:860) [0m > [31m at org.scalatest.tools.Runner.main(Runner.scala) [0m > [31m ... [0m > [31m Cause: org.apache.phoenix.exception.PhoenixParserException: ERROR 601 > (42P00): Syntax error. Encountered "" at line 1, column 62. [0m > [31m at > org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:78) [0m > [31m ... [0m > [31m Cause: org.antlr.runtime.NoViableAltException: [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:683) > [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:474) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m ... [0m > [ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-1877) Spark end2end tests are breaking the build
[ https://issues.apache.org/jira/browse/PHOENIX-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499082#comment-14499082 ] Josh Mahonin edited comment on PHOENIX-1877 at 4/17/15 1:07 AM: It appears that commit e1bbb944126da52d14bf5df580167644a1cd8499 caused the PhoenixSparkIT tests to fail. The new ASL license at the top of setup.sql doesn't work properly with the way the test loads and and executes each line as a JDBC statement. I've attached a patch that gets the test passing again by filtering on comments and blank lines in the setup.sql file. was (Author: jmahonin): It appears that commit e1bbb944126da52d14bf5df580167644a1cd8499 caused the PhoenixSparkIT tests to fail. The new ASL license at the top of setup.sql doesn't work properly with the way the setup.sql file gets loaded and and executes each line as a JDBC statement. I've attached a patch that gets the test passing again by filtering on comments and blank lines in the setup.sql file. > Spark end2end tests are breaking the build > -- > > Key: PHOENIX-1877 > URL: https://issues.apache.org/jira/browse/PHOENIX-1877 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-1877.patch > > > See below for the error. Maybe we should disable these tests until this can > be sorted out? > [31m*** RUN ABORTED *** [0m > [31m java.util.concurrent.ExecutionException: > org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): > Syntax error. Encountered "" at line 1, column 62. [0m > [31m at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > [0m > [31m at java.util.concurrent.FutureTask.get(FutureTask.java:111) [0m > [31m at > org.scalatest.tools.ConcurrentDistributor.waitUntilDone(ConcurrentDistributor.scala:52) > [0m > [31m at > org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2549) [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044) > [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043) > [0m > [31m at > org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722) > [0m > [31m at > org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043) > [0m > [31m at org.scalatest.tools.Runner$.main(Runner.scala:860) [0m > [31m at org.scalatest.tools.Runner.main(Runner.scala) [0m > [31m ... [0m > [31m Cause: org.apache.phoenix.exception.PhoenixParserException: ERROR 601 > (42P00): Syntax error. Encountered "" at line 1, column 62. [0m > [31m at > org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:78) [0m > [31m ... [0m > [31m Cause: org.antlr.runtime.NoViableAltException: [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:683) > [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:474) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m ... [0
[jira] [Commented] (PHOENIX-1877) Spark end2end tests are breaking the build
[ https://issues.apache.org/jira/browse/PHOENIX-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499082#comment-14499082 ] Josh Mahonin commented on PHOENIX-1877: --- It appears that commit e1bbb944126da52d14bf5df580167644a1cd8499 caused the PhoenixSparkIT tests to fail. The new ASL license at the top of setup.sql doesn't work properly with the way the setup.sql file gets loaded and and executes each line as a JDBC statement. I've attached a patch that gets the test passing again by filtering on comments and blank lines in the setup.sql file. > Spark end2end tests are breaking the build > -- > > Key: PHOENIX-1877 > URL: https://issues.apache.org/jira/browse/PHOENIX-1877 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > > See below for the error. Maybe we should disable these tests until this can > be sorted out? > [31m*** RUN ABORTED *** [0m > [31m java.util.concurrent.ExecutionException: > org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): > Syntax error. Encountered "" at line 1, column 62. [0m > [31m at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > [0m > [31m at java.util.concurrent.FutureTask.get(FutureTask.java:111) [0m > [31m at > org.scalatest.tools.ConcurrentDistributor.waitUntilDone(ConcurrentDistributor.scala:52) > [0m > [31m at > org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2549) [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044) > [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043) > [0m > [31m at > org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722) > [0m > [31m at > org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043) > [0m > [31m at org.scalatest.tools.Runner$.main(Runner.scala:860) [0m > [31m at org.scalatest.tools.Runner.main(Runner.scala) [0m > [31m ... [0m > [31m Cause: org.apache.phoenix.exception.PhoenixParserException: ERROR 601 > (42P00): Syntax error. Encountered "" at line 1, column 62. [0m > [31m at > org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:78) [0m > [31m ... [0m > [31m Cause: org.antlr.runtime.NoViableAltException: [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:683) > [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:474) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m ... [0m > [ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1877) Spark end2end tests are breaking the build
[ https://issues.apache.org/jira/browse/PHOENIX-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499727#comment-14499727 ] Josh Mahonin commented on PHOENIX-1877: --- Although the Jenkins job doesn't seem to run verify on the 4.x-Hbase-0.98 branch, the same issue exists there as well. > Spark end2end tests are breaking the build > -- > > Key: PHOENIX-1877 > URL: https://issues.apache.org/jira/browse/PHOENIX-1877 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-1877.patch > > > See below for the error. Maybe we should disable these tests until this can > be sorted out? > [31m*** RUN ABORTED *** [0m > [31m java.util.concurrent.ExecutionException: > org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): > Syntax error. Encountered "" at line 1, column 62. [0m > [31m at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > [0m > [31m at java.util.concurrent.FutureTask.get(FutureTask.java:111) [0m > [31m at > org.scalatest.tools.ConcurrentDistributor.waitUntilDone(ConcurrentDistributor.scala:52) > [0m > [31m at > org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2549) [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044) > [0m > [31m at > org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043) > [0m > [31m at > org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722) > [0m > [31m at > org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043) > [0m > [31m at org.scalatest.tools.Runner$.main(Runner.scala:860) [0m > [31m at org.scalatest.tools.Runner.main(Runner.scala) [0m > [31m ... [0m > [31m Cause: org.apache.phoenix.exception.PhoenixParserException: ERROR 601 > (42P00): Syntax error. Encountered "" at line 1, column 62. [0m > [31m at > org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:78) [0m > [31m ... [0m > [31m Cause: org.antlr.runtime.NoViableAltException: [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:683) > [0m > [31m at > org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:474) > [0m > [31m at > org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108) [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:1005) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1086) > [0m > [31m at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1149) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80) > [0m > [31m at > org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78) > [0m > [31m at scala.collection.Iterator$class.foreach(Iterator.scala:727) [0m > [31m at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) [0m > [31m ... [0m > [ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1916) Spark tests are failing
[ https://issues.apache.org/jira/browse/PHOENIX-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511607#comment-14511607 ] Josh Mahonin commented on PHOENIX-1916: --- I suspect this is a result of PHOENIX-1682. I've attached patch removes the quoting around the table name when setting it. The tests pass for me. FYI, I'll be going off grid on vacation for a few weeks. If anything comes up with the phoenix-spark module, it's not that I'm not interested, but may be unavailable for response until I get back. > Spark tests are failing > --- > > Key: PHOENIX-1916 > URL: https://issues.apache.org/jira/browse/PHOENIX-1916 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain > > Spark tests have been failing on the master branch. [~jmahonin], > [~maghamraviki...@gmail.com] - can one of you take a look. > Sample fail run: https://builds.apache.org/job/Phoenix-master/716/console -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1916) Spark tests are failing
[ https://issues.apache.org/jira/browse/PHOENIX-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1916: -- Attachment: PHOENIX-1916.patch > Spark tests are failing > --- > > Key: PHOENIX-1916 > URL: https://issues.apache.org/jira/browse/PHOENIX-1916 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain > Attachments: PHOENIX-1916.patch > > > Spark tests have been failing on the master branch. [~jmahonin], > [~maghamraviki...@gmail.com] - can one of you take a look. > Sample fail run: https://builds.apache.org/job/Phoenix-master/716/console -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-1916) Spark tests are failing
[ https://issues.apache.org/jira/browse/PHOENIX-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511607#comment-14511607 ] Josh Mahonin edited comment on PHOENIX-1916 at 4/24/15 7:40 PM: I suspect this is a result of PHOENIX-1682. I've attached patch removes the quoting around the table name when setting it. The tests pass for me. FYI, I'll be going off the grid on vacation for a few weeks. If anything comes up with the phoenix-spark module, it's not that I'm not interested, but may be unavailable for response until I get back. was (Author: jmahonin): I suspect this is a result of PHOENIX-1682. I've attached patch removes the quoting around the table name when setting it. The tests pass for me. FYI, I'll be going off grid on vacation for a few weeks. If anything comes up with the phoenix-spark module, it's not that I'm not interested, but may be unavailable for response until I get back. > Spark tests are failing > --- > > Key: PHOENIX-1916 > URL: https://issues.apache.org/jira/browse/PHOENIX-1916 > Project: Phoenix > Issue Type: Bug >Reporter: Samarth Jain > Attachments: PHOENIX-1916.patch > > > Spark tests have been failing on the master branch. [~jmahonin], > [~maghamraviki...@gmail.com] - can one of you take a look. > Sample fail run: https://builds.apache.org/job/Phoenix-master/716/console -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1924) PhoenixSparkIT failing in 4.x-HBase-0.98
[
https://issues.apache.org/jira/browse/PHOENIX-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14513253#comment-14513253
]
Josh Mahonin commented on PHOENIX-1924:
---
This looks like the same error as PHOENIX-1817. Did that patch make it on this
branch? On mobile on airport wifi, I can't check much further.
> PhoenixSparkIT failing in 4.x-HBase-0.98
>
>
> Key: PHOENIX-1924
> URL: https://issues.apache.org/jira/browse/PHOENIX-1924
> Project: Phoenix
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Josh Mahonin
>
> https://builds.apache.org/view/All/job/Phoenix-4.x-HBase-0.98/ws/phoenix-spark/target/surefire-reports/TestSuite.txt
> {noformat}
> Discovery starting.
> Discovery completed in 342 milliseconds.
> Run starting. Expected test count is: 17
> PhoenixSparkIT:
> *** RUN ABORTED *** (19 seconds, 223 milliseconds)
> java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00):
> Syntax error. Encountered "" at line 1, column 62.
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at
> org.scalatest.tools.ConcurrentDistributor.waitUntilDone(ConcurrentDistributor.scala:52)
> at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2549)
> at
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
> at
> org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
> at
> org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
> at
> org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
> at org.scalatest.tools.Runner$.main(Runner.scala:860)
> at org.scalatest.tools.Runner.main(Runner.scala)
> Cause: org.apache.phoenix.exception.PhoenixParserException: ERROR 601
> (42P00): Syntax error. Encountered "" at line 1, column 62.
> at
> org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:111)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:949)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1030)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1093)
> at
> org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80)
> at
> org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:78)
> at
> org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
> at
> org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:48)
> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
> at org.apache.phoenix.spark.PhoenixSparkIT.run(PhoenixSparkIT.scala:48)
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Cause: org.antlr.runtime.NoViableAltException:
> at
> org.apache.phoenix.parse.PhoenixSQLParser.oneStatement(PhoenixSQLParser.java:674)
> at
> org.apache.phoenix.parse.PhoenixSQLParser.statement(PhoenixSQLParser.java:473)
> at org.apache.phoenix.parse.SQLParser.parseStatement(SQLParser.java:108)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$PhoenixStatementParser.parseStatement(PhoenixStatement.java:949)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.parseStatement(PhoenixStatement.java:1030)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1093)
> at
> org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:80)
> at
> org.apache.phoenix.spark.PhoenixSparkIT$$anonfun$beforeAll$1.apply(PhoenixSparkIT.scala:78)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> org.apache.phoenix.spark.PhoenixSparkIT.beforeAll(PhoenixSparkIT.scala:78)
> at
> org.scalatest.BeforeAndAfterAll
[jira] [Commented] (PHOENIX-1926) Attempt to update a record in HBase via Phoenix from a Spark job causes java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access
[ https://issues.apache.org/jira/browse/PHOENIX-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515878#comment-14515878 ] Josh Mahonin commented on PHOENIX-1926: --- Hi Dmitri, On mobile so pardon the brevity. Both the spark driver and spark executors need to have the hbase-protocol JAR in the class path. I've seen those same tickets, and I'm not sure why it's still required, but it is. I think you can do this by rolling all of the Phoenix dependencies into your job JAR, but I've had success by modifying the SPARK_CLASSPATH to include it. That method is deprecated, so there's also spark conf settings to set the driver and executor class paths as well. These days, I just include the full phoenix-client JAR in the spark class path, and don't package anything platform related in the jobs JAR. It's a bit of a sledgehammer solution, but it works for me. Good luck! > Attempt to update a record in HBase via Phoenix from a Spark job causes > java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > -- > > Key: PHOENIX-1926 > URL: https://issues.apache.org/jira/browse/PHOENIX-1926 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.3.1 > Environment: centos x86_64 GNU/Linux > Phoenix: 4.3.1 custom built for HBase 0.98.9, Hadoop 2.4.0 > HBase: 0.98.9-hadoop2 > Hadoop: 2.4.0 > Spark: spark-1.3.0-bin-hadoop2.4 >Reporter: Dmitry Goldenberg >Priority: Critical > > Performing an UPSERT from within a Spark job, > UPSERT INTO items(entry_id, prop1, pop2) VALUES(?, ?, ?) > causes > 15/04/26 18:22:06 WARN client.HTable: Error calling coprocessor service > org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for > row \x00\x00ITEMS > java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1007) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307) > at > org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:237) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:231) > at > org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:207) > at > org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:248) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:503) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:494) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:288) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:287) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:219) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:174) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:179) > ... > Caused by: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
[jira] [Commented] (PHOENIX-1926) Attempt to update a record in HBase via Phoenix from a Spark job causes java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access
[ https://issues.apache.org/jira/browse/PHOENIX-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517423#comment-14517423 ] Josh Mahonin commented on PHOENIX-1926: --- +1 to more docs. I had written up some in a PR for the site on svn, and I believe I updated the README.md in another PR on github. @ravi may have a better idea of where those tickets are at, I'm about to hop on a plane. Sorry for the lack of updates here, will be able to communicate more effectively after I'm back from vacation on May 8. > Attempt to update a record in HBase via Phoenix from a Spark job causes > java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > -- > > Key: PHOENIX-1926 > URL: https://issues.apache.org/jira/browse/PHOENIX-1926 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.3.1 > Environment: centos x86_64 GNU/Linux > Phoenix: 4.3.1 custom built for HBase 0.98.9, Hadoop 2.4.0 > HBase: 0.98.9-hadoop2 > Hadoop: 2.4.0 > Spark: spark-1.3.0-bin-hadoop2.4 >Reporter: Dmitry Goldenberg >Priority: Critical > > Performing an UPSERT from within a Spark job, > UPSERT INTO items(entry_id, prop1, pop2) VALUES(?, ?, ?) > causes > 15/04/26 18:22:06 WARN client.HTable: Error calling coprocessor service > org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for > row \x00\x00ITEMS > java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1007) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307) > at > org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:237) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:231) > at > org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:207) > at > org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:248) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:503) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:494) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:288) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:287) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:219) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:174) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:179) > ... > Caused by: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >
[jira] [Created] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
Josh Mahonin created PHOENIX-1968: - Summary: Phoenix-Spark: Should support saving arrays Key: PHOENIX-1968 URL: https://issues.apache.org/jira/browse/PHOENIX-1968 Project: Phoenix Issue Type: Bug Reporter: Josh Mahonin Assignee: Josh Mahonin At present, we only use 'setObject' on the PreparedStatement when writing back to Phoenix. This patch allows calling 'setArray' with the appropriate sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1968: -- Attachment: PHOENIX-1968.patch Applies cleanly on master > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-1968.patch > > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[
https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14542306#comment-14542306
]
Josh Mahonin commented on PHOENIX-1968:
---
Hmm, I just noticed that I've slightly altered the previous statement:
{code}
statement.setObject(i + 1, finalObj, finalType)
{code}
to
{code}
statement.setObject(i + 1, finalObj)
{code}
The unit tests still pass though, does anyone know if there are any cases where
setting the sql type is explicitly required?
> Phoenix-Spark: Should support saving arrays
> ---
>
> Key: PHOENIX-1968
> URL: https://issues.apache.org/jira/browse/PHOENIX-1968
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Mahonin
>Assignee: Josh Mahonin
> Attachments: PHOENIX-1968.patch
>
>
> At present, we only use 'setObject' on the PreparedStatement when writing
> back to Phoenix. This patch allows calling 'setArray' with the appropriate
> sql base type when an array is passed in as a column value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-1816) Add markdown page on Phoenix website describing how to use Spark integration
[ https://issues.apache.org/jira/browse/PHOENIX-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1816: -- Attachment: (was: README.md) > Add markdown page on Phoenix website describing how to use Spark integration > > > Key: PHOENIX-1816 > URL: https://issues.apache.org/jira/browse/PHOENIX-1816 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.4.0 > > > Now that we have some nifty integration with Spark, we should add a markdown > page to our site (http://phoenix.apache.org/), with a link in the Using menu > that describes how to use it. > See http://phoenix.apache.org/building_website.html for how to modify the the > website (in lives in svn). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1816) Add markdown page on Phoenix website describing how to use Spark integration
[ https://issues.apache.org/jira/browse/PHOENIX-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1816: -- Attachment: README.md > Add markdown page on Phoenix website describing how to use Spark integration > > > Key: PHOENIX-1816 > URL: https://issues.apache.org/jira/browse/PHOENIX-1816 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.4.0 > > Attachments: README.md > > > Now that we have some nifty integration with Spark, we should add a markdown > page to our site (http://phoenix.apache.org/), with a link in the Using menu > that describes how to use it. > See http://phoenix.apache.org/building_website.html for how to modify the the > website (in lives in svn). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1816) Add markdown page on Phoenix website describing how to use Spark integration
[ https://issues.apache.org/jira/browse/PHOENIX-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581942#comment-14581942 ] Josh Mahonin commented on PHOENIX-1816: --- Updated this PR to merge to master with updated Spark documentation. The updated instructions include adding the client JAR to the spark classpath, instead of attempting to bundle in a spark jobs JAR. Also attached the README.md, in case we want to update the website as well. > Add markdown page on Phoenix website describing how to use Spark integration > > > Key: PHOENIX-1816 > URL: https://issues.apache.org/jira/browse/PHOENIX-1816 > Project: Phoenix > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.4.0 > > Attachments: README.md > > > Now that we have some nifty integration with Spark, we should add a markdown > page to our site (http://phoenix.apache.org/), with a link in the Using menu > that describes how to use it. > See http://phoenix.apache.org/building_website.html for how to modify the the > website (in lives in svn). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581948#comment-14581948 ] Josh Mahonin commented on PHOENIX-1968: --- [~maghamraviki...@gmail.com] Any updates on this? Thanks! > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-1968.patch > > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1968: -- Attachment: (was: PHOENIX-1968.patch) > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1968: -- Attachment: PHOENIX-1968.patch Updated patch for current master > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-1968.patch > > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582239#comment-14582239 ] Josh Mahonin edited comment on PHOENIX-1968 at 6/11/15 5:13 PM: Updated patch for current master. It looks like the old one wasn't applying cleanly - not sure why. Despite some weird output from 'mvn verify' (RpcServer.reader output), the tests pass. was (Author: jmahonin): Updated patch for current master > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-1968.patch > > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2032) psql.py is broken after PHOENIX-2013
[
https://issues.apache.org/jira/browse/PHOENIX-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14584633#comment-14584633
]
Josh Mahonin commented on PHOENIX-2032:
---
It would be nice if Spark users could just use the regular client JAR with
phoenix-spark included in it, by simply adding it to the Spark classpath,
although a spark-specific JAR wouldn't be too terrible either if that's the
only option.
However, JAR size issue I think is my fault. Looking at the phoenix-spark
pom.xml, scala-library, spark-core or spark-sql should all be marked as scope
'provided', since Spark offers these it in its own classpath.
The other dependencies not marked as test or provided, hadoop-client,
hadoop-common, hbase-client, I think are all included in the client JAR anyway,
so I don't think those would matter, but I may be wrong.
On Monday, I'll try adjust those provided settings and see if everything is
working for me. If that reduces the JAR size, would we be able to get
phoenix-spark back in the client JAR?
> psql.py is broken after PHOENIX-2013
>
>
> Key: PHOENIX-2032
> URL: https://issues.apache.org/jira/browse/PHOENIX-2032
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.5.0, 4.4.1
>Reporter: Sergio Peleato
>Assignee: Nick Dimiduk
>Priority: Blocker
> Fix For: 5.0.0, 4.5.0, 4.4.1
>
> Attachments: 2032.opt2a.diff, 2032.opt2a.diff, 2032.patch,
> PHOENIX-2032.00.patch
>
>
> psql is no longer able to load the phoenix driver after PHOENIX-2013.
> {noformat}
> $ ./bin/psql.py localhost examples/WEB_STAT.sql
>
>
> java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost
> at java.sql.DriverManager.getConnection(DriverManager.java:596)
> at java.sql.DriverManager.getConnection(DriverManager.java:187)
> at
> org.apache.phoenix.util.PhoenixRuntime.main(PhoenixRuntime.java:183)
> {noformat}
> Hat-tip to [~chrajeshbab...@gmail.com] for tracking down the breaking change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-2040) Mark spark/scala dependencies as 'provided'
Josh Mahonin created PHOENIX-2040: - Summary: Mark spark/scala dependencies as 'provided' Key: PHOENIX-2040 URL: https://issues.apache.org/jira/browse/PHOENIX-2040 Project: Phoenix Issue Type: Bug Reporter: Josh Mahonin Assignee: Josh Mahonin The Spark runtime provides both the scala library, as well as the Spark dependencies, so these should be marked as 'provided' in the phoenix-spark module. This greatly reduces the size of the resulting client JAR. This patch also adds back phoenix-spark to the list of modules in the assembly JAR, to be included in the client JAR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2040) Mark spark/scala dependencies as 'provided'
[ https://issues.apache.org/jira/browse/PHOENIX-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2040: -- Attachment: PHOENIX-2040.patch > Mark spark/scala dependencies as 'provided' > --- > > Key: PHOENIX-2040 > URL: https://issues.apache.org/jira/browse/PHOENIX-2040 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2040.patch > > > The Spark runtime provides both the scala library, as well as the Spark > dependencies, so these should be marked as 'provided' in the phoenix-spark > module. This greatly reduces the size of the resulting client JAR. > This patch also adds back phoenix-spark to the list of modules in the > assembly JAR, to be included in the client JAR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2040) Mark spark/scala dependencies as 'provided'
[
https://issues.apache.org/jira/browse/PHOENIX-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586080#comment-14586080
]
Josh Mahonin commented on PHOENIX-2040:
---
Resulting file sizes after patch:
{noformat}
ls -lh phoenix-assembly/target/
total 648648
drwxr-xr-x 2 joshmahonin staff68B 15 Jun 09:59 archive-tmp
-rw-r--r-- 1 joshmahonin staff11K 15 Jun 09:59 checkstyle-checker.xml
-rw-r--r-- 1 joshmahonin staff 798B 15 Jun 09:59 checkstyle-header.txt
-rw-r--r-- 1 joshmahonin staff80B 15 Jun 09:59 checkstyle-result.xml
-rw-r--r-- 1 joshmahonin staff 2.0K 15 Jun 09:59 checkstyle-suppressions.xml
drwxr-xr-x 3 joshmahonin staff 102B 15 Jun 09:59 maven-archiver
drwxr-xr-x 3 joshmahonin staff 102B 15 Jun 09:59
maven-shared-archive-resources
-rw-r--r-- 1 joshmahonin staff 4.3M 15 Jun 09:59
phoenix-4.5.0-SNAPSHOT-client-minimal.jar
-rw-r--r-- 1 joshmahonin staff28M 15 Jun 09:59
phoenix-4.5.0-SNAPSHOT-client-without-hbase.jar
-rw-r--r-- 1 joshmahonin staff65M 15 Jun 09:59
phoenix-4.5.0-SNAPSHOT-client.jar
-rw-r--r-- 1 joshmahonin staff 4.4M 15 Jun 09:59
phoenix-4.5.0-SNAPSHOT-server-without-antlr.jar
-rw-r--r-- 1 joshmahonin staff 5.7M 15 Jun 09:59
phoenix-4.5.0-SNAPSHOT-server.jar
-rw-r--r-- 1 joshmahonin staff 1.7M 15 Jun 10:00
phoenix-4.5.0-SNAPSHOT-source.tar.gz
-rw-r--r-- 1 joshmahonin staff 208M 15 Jun 10:00
phoenix-4.5.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 joshmahonin staff 2.8K 15 Jun 09:59
phoenix-assembly-4.5.0-SNAPSHOT-tests.jar
{noformat}
> Mark spark/scala dependencies as 'provided'
> ---
>
> Key: PHOENIX-2040
> URL: https://issues.apache.org/jira/browse/PHOENIX-2040
> Project: Phoenix
> Issue Type: Bug
>Reporter: Josh Mahonin
>Assignee: Josh Mahonin
> Attachments: PHOENIX-2040.patch
>
>
> The Spark runtime provides both the scala library, as well as the Spark
> dependencies, so these should be marked as 'provided' in the phoenix-spark
> module. This greatly reduces the size of the resulting client JAR.
> This patch also adds back phoenix-spark to the list of modules in the
> assembly JAR, to be included in the client JAR.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2040) Mark spark/scala dependencies as 'provided'
[ https://issues.apache.org/jira/browse/PHOENIX-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586618#comment-14586618 ] Josh Mahonin commented on PHOENIX-2040: --- [~ndimiduk] I've applied the patch to branch 4.4-HBase-0.98 (4.4.1-Hbase-0.98-SNAPSHOT) and can confirm that both sqlline and psql work, though the benign sqlline NPE error I've seen in 4.4.0 is still there. Re: Spark jobs and client JAR... that one's tricky. Myself, and other users [1] have run into Spark classpath issues when working with HBase. Although it should be technically possible to manually include in a jobs JAR all of the dependencies necessary to get Spark talking to Phoenix, in practice it's fraught with problems. Since most Spark distributions already include some HBase libs, the classpath generally has to get updated with the proper 'hbase-protocol' JAR no matter what. What I've found to be the easiest solution, is just to include the Phoenix client JAR in the Spark classpath for both the driver and executors and forget about it [2]. I don't necessarily think that's the best way to do it, but it's the only one I've found that a) works, and b) keeps the pom.xml and build.sbt files manageable. I am sort of hoping that in time some other smart folks (maybe Hortonworks or Cloudera?) who have a bit more experience with Spark and classpath issues can help out here. [1] http://mail-archives.apache.org/mod_mbox/phoenix-user/201506.mbox/browser [2] https://phoenix.apache.org/phoenix_spark.html > Mark spark/scala dependencies as 'provided' > --- > > Key: PHOENIX-2040 > URL: https://issues.apache.org/jira/browse/PHOENIX-2040 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.5.0 > > Attachments: PHOENIX-2040.patch > > > The Spark runtime provides both the scala library, as well as the Spark > dependencies, so these should be marked as 'provided' in the phoenix-spark > module. This greatly reduces the size of the resulting client JAR. > This patch also adds back phoenix-spark to the list of modules in the > assembly JAR, to be included in the client JAR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-2040) Mark spark/scala dependencies as 'provided'
[ https://issues.apache.org/jira/browse/PHOENIX-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586618#comment-14586618 ] Josh Mahonin edited comment on PHOENIX-2040 at 6/15/15 8:21 PM: [~ndimiduk] I've applied the patch to branch 4.4-HBase-0.98 (4.4.1-Hbase-0.98-SNAPSHOT) and can confirm that both sqlline and psql work, though the benign sqlline NPE error I've seen in 4.4.0 is still there. Re: Spark jobs and client JAR... that one's tricky. Myself, and other users [1] have run into Spark classpath issues when working with HBase. Although it should be technically possible to manually include in a jobs JAR all of the dependencies necessary to get Spark talking to Phoenix, in practice it's fraught with problems. Since most Spark distributions already include some HBase libs, the classpath generally has to get updated with the proper 'hbase-protocol' JAR no matter what. What I've found to be the easiest solution, is just to include the Phoenix client JAR in the Spark classpath for both the driver and executors and forget about it [2]. I don't necessarily think that's the best way to do it, but it's the only one I've found that a) works, and b) keeps the pom.xml and build.sbt files manageable. I am sort of hoping that in time some other smart folks (maybe Hortonworks or Cloudera?) who have a bit more experience with Spark and classpath issues can help out here. [1] http://mail-archives.apache.org/mod_mbox/phoenix-user/201506.mbox/%3C2398402.1exqR2HZ3A%40madara%3E [2] https://phoenix.apache.org/phoenix_spark.html was (Author: jmahonin): [~ndimiduk] I've applied the patch to branch 4.4-HBase-0.98 (4.4.1-Hbase-0.98-SNAPSHOT) and can confirm that both sqlline and psql work, though the benign sqlline NPE error I've seen in 4.4.0 is still there. Re: Spark jobs and client JAR... that one's tricky. Myself, and other users [1] have run into Spark classpath issues when working with HBase. Although it should be technically possible to manually include in a jobs JAR all of the dependencies necessary to get Spark talking to Phoenix, in practice it's fraught with problems. Since most Spark distributions already include some HBase libs, the classpath generally has to get updated with the proper 'hbase-protocol' JAR no matter what. What I've found to be the easiest solution, is just to include the Phoenix client JAR in the Spark classpath for both the driver and executors and forget about it [2]. I don't necessarily think that's the best way to do it, but it's the only one I've found that a) works, and b) keeps the pom.xml and build.sbt files manageable. I am sort of hoping that in time some other smart folks (maybe Hortonworks or Cloudera?) who have a bit more experience with Spark and classpath issues can help out here. [1] http://mail-archives.apache.org/mod_mbox/phoenix-user/201506.mbox/browser [2] https://phoenix.apache.org/phoenix_spark.html > Mark spark/scala dependencies as 'provided' > --- > > Key: PHOENIX-2040 > URL: https://issues.apache.org/jira/browse/PHOENIX-2040 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.5.0 > > Attachments: PHOENIX-2040.patch > > > The Spark runtime provides both the scala library, as well as the Spark > dependencies, so these should be marked as 'provided' in the phoenix-spark > module. This greatly reduces the size of the resulting client JAR. > This patch also adds back phoenix-spark to the list of modules in the > assembly JAR, to be included in the client JAR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601483#comment-14601483 ] Josh Mahonin commented on PHOENIX-1659: --- I've done some work with metadata compatibility with other JDBC tooling, but haven't run into any with the REMARKS column before. Looking at: http://docs.oracle.com/javase/7/docs/api/java/sql/DatabaseMetaData.html It specifies that the 'REMARKS' column may be null. Is it possible this is a bug in MyBatis, expecting this field to always be present? > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1659: -- Attachment: PHOENIX-1659.patch Add the 'REMARKS' column to the getColumns() call with a unit test in QueryDatabaseMetaDataIT.java. > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > Attachments: PHOENIX-1659.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602870#comment-14602870 ] Josh Mahonin commented on PHOENIX-1659: --- This passes the unit tests and the QueryDatabaseMetaDataIT integration test for me. My machine got through 213 tests in the integration suite before going to sleep on me, but it hasn't had a full run yet. > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > Attachments: PHOENIX-1659.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1659: -- Attachment: (was: PHOENIX-1659.patch) > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1659: -- Attachment: PHOENIX-1659-v2.patch Updated patch with tests to ensure lookup by column name / position return null instead of throwing exceptions > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > Attachments: PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605874#comment-14605874 ] Josh Mahonin commented on PHOENIX-1659: --- Pushed to master, 4.4 and 4.x branches. Hopefully I didn't mess anything up! > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > Attachments: PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606171#comment-14606171 ] Josh Mahonin commented on PHOENIX-1659: --- It looks like there's an issue with the flume integration, and reverting locally indicates it is introduced by this patch. I'm digging through now, but I don't see anything obvious sticking out yet. If anyone a bit more familiar with the module has any suggestions, I'm all ears. > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > Attachments: PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1659: -- Attachment: PHOENIX-1659-followup.patch Adjust COLUMN_FAMILY_POSITION in QueryUtil to account for new REMARKS column. Both org.apache.phoenix.flume.serializer.BaseEventSerializer and org.apache.phoenix.util.CSVCommonsLoader referred to the field. Integration tests for each component are passing with this patch. > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie > Labels: Newbie > Attachments: PHOENIX-1659-followup.patch, PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (PHOENIX-1913) Unable to build the website code in svn
[ https://issues.apache.org/jira/browse/PHOENIX-1913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin reopened PHOENIX-1913: --- I just ran into the same issue, it doesn't work with Java 8, despite reporting a build success. The errors displayed are the same as in this report. James asked me to re-open and assign to Mujtaba. > Unable to build the website code in svn > --- > > Key: PHOENIX-1913 > URL: https://issues.apache.org/jira/browse/PHOENIX-1913 > Project: Phoenix > Issue Type: Bug >Reporter: maghamravikiran >Assignee: Mujtaba Chohan > > Following the steps mentioned in > http://phoenix.apache.org/building_website.html I get the below exception > Generate Phoenix Website > Pre-req: On source repo run $ mvn install -DskipTests > BUILDING LANGUAGE REFERENCE > === > src/tools/org/h2/build/BuildBase.java:136: error: no suitable method found > for replaceAll(String,String,String) > pattern = replaceAll(pattern, "/", File.separator); > ^ > method List.replaceAll(UnaryOperator) is not applicable > (actual and formal argument lists differ in length) > method ArrayList.replaceAll(UnaryOperator) is not applicable > (actual and formal argument lists differ in length) > 1 error > Error: Could not find or load main class org.h2.build.Build > BUILDING SITE > === > [INFO] Scanning for projects... > [ERROR] The build could not read 1 project -> [Help 1] > [ERROR] > [ERROR] The project org.apache.phoenix:phoenix-site:[unknown-version] > (/Users/ravimagham/git/sources/phoenix/site/source/pom.xml) has 1 error > [ERROR] Non-resolvable parent POM: Could not find artifact > org.apache.phoenix:phoenix:pom:4.4.0-SNAPSHOT and 'parent.relativePath' > points at wrong local POM @ line 4, column 11 -> [Help 2] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException > [ERROR] [Help 2] > http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException > Can you please have a look ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611320#comment-14611320 ] Josh Mahonin commented on PHOENIX-2088: --- Just a note that this will affect the phoenix-spark module here: https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/ConfigurationUtil.scala#L55-L64 I believe the motivation behind that code was to recreate the ColumnInfo objects from a serializable type, and there was a convenient utility method there to provide that. Does the mapreduce integration not need that capability any more? I suspect whatever works for mapreduce can be made to work with spark, but if you see the 'PhoenixRecordWritable' class, it uses the ColumnInfo object to derive the types for writing: https://github.com/apache/phoenix/blob/master/phoenix-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRecordWritable.scala#L33-L74 > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva > Attachments: PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611410#comment-14611410 ] Josh Mahonin commented on PHOENIX-2088: --- On a first look, I'd vote for another way of serializing the names / column infos, if possible. However, it's been a while since I was knee-deep in that code. I recall the main issue being that neither the 'Configuration', nor 'ColumnInfo' objects are serializable, hence the call to the encode / decode functions. I haven't had a chance to really dive into it yet, but I suspect that these same issues are shared with mapreduce / pig as well, since that's where most of the spark code came from to begin with. I imagine there's probably a general solution that should work for all of them, but if these changes are already compatible with mapreduce and pig, it should be possible to adjust the spark integration accordingly. Am willing and able to help where needed. I'll try take a deeper dive tomorrow to confirm my assumptions. > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2088: -- Attachment: PHOENIX-2088-wip-v2.patch Additions to previous patch for phoenix-spark module Instead of relying on the ColumnInfoEncoderDecoder, I've tweaked the usage of the Configuration and ColumnInfo objects so that they're never serialized, but instead recreated / derived on a per-partition basis. This passes the phoenix-spark integration tests, but the phoenix-pig unit tests are failing, though I don't think that was caused by my additions. > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-2088-wip-v2.patch, PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin reassigned PHOENIX-1659: - Assignee: Josh Mahonin > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie >Assignee: Josh Mahonin > Labels: Newbie > Attachments: PHOENIX-1659-followup.patch, PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612465#comment-14612465 ] Josh Mahonin commented on PHOENIX-1659: --- What's the protocol for resolving? These changes are pushed in, but the builds and build server, have been flaky for a while. Am I OK to resolve this, or should I park it and wait until we get some clean builds out? > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie >Assignee: Josh Mahonin > Labels: Newbie > Attachments: PHOENIX-1659-followup.patch, PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-1659) PhoenixDatabaseMetaData.getColumns does not return REMARKS column
[ https://issues.apache.org/jira/browse/PHOENIX-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin resolved PHOENIX-1659. --- Resolution: Fixed Fix Version/s: 4.4.1 4.5.0 5.0.0 > PhoenixDatabaseMetaData.getColumns does not return REMARKS column > - > > Key: PHOENIX-1659 > URL: https://issues.apache.org/jira/browse/PHOENIX-1659 > Project: Phoenix > Issue Type: Improvement >Reporter: Kevin Guthrie >Assignee: Josh Mahonin > Labels: Newbie > Fix For: 5.0.0, 4.5.0, 4.4.1 > > Attachments: PHOENIX-1659-followup.patch, PHOENIX-1659-v2.patch > > > I am attempting to use phoenix as a jdbc input for MyBatis, and I am getting > errors when loading the meta data for table columns because the value of the > REMARKS column is not returned by the query issued in > PhoenixDatabaseMetaData.getColumns -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612494#comment-14612494 ] Josh Mahonin commented on PHOENIX-2088: --- I'm just reading through this thread again and I should clarify that this updated patch is only meant to prove that it's possible to use the phoenix-spark integration without the ColumnInfoEncoderDecoder, if that's necessary for the full solution. All of the spark-specific column / delimiter stuff works pretty much the same as the Pig / MapReduce integration, so whatever the right fix is that works for both of them should be fine for Spark too. > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-2088-wip-v2.patch, PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2065) Throw TableNotFoundException when select all columns of one column family from the table with schema
[ https://issues.apache.org/jira/browse/PHOENIX-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613213#comment-14613213 ] Josh Mahonin commented on PHOENIX-2065: --- I can't compile after this patch. Looks like maybe a missing import in QueryCompilerTest? > Throw TableNotFoundException when select all columns of one column family > from the table with schema > > > Key: PHOENIX-2065 > URL: https://issues.apache.org/jira/browse/PHOENIX-2065 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.4.0 >Reporter: Jun Ng > Fix For: 5.0.0, 4.5.0, 4.4.1 > > Attachments: PHOENIX-2065-1.patch, PHOENIX-2065-2.patch, > PHOENIX-2065-for-4.4-HBase-1.0.patch, PHOENIX-2065-for-master.patch, > PHOENIX-2065.patch > > > 1、create table with schema, like "create table s.t (k integer not null > primary key, f1.v1 varchar, f2.v2 varchar);" > 2、select table like "select f1.* from s.t;" > It goes wrong, throws TableNotFoundException: > 0: jdbc:phoenix:160.149.0.117> select a.* from dept.saler; > Error: ERROR 1012 (42M03): Table undefined. tableName=SALER > (state=42M03,code=1012) > org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table > undefined. tableName=SALER > at > org.apache.phoenix.compile.FromCompiler$MultiTableColumnResolver.resolveTable(FromCompiler.java:690) > at > org.apache.phoenix.compile.FromCompiler$MultiTableColumnResolver.resolveColumnFamily(FromCompiler.java:713) > at > org.apache.phoenix.compile.FromCompiler$MultiTableColumnResolver.resolveColumn(FromCompiler.java:745) > at > org.apache.phoenix.compile.FromCompiler$ProjectedTableColumnResolver.resolveColumn(FromCompiler.java:796) > at > org.apache.phoenix.compile.ProjectionCompiler.projectTableColumnFamily(ProjectionCompiler.java:266) > at > org.apache.phoenix.compile.ProjectionCompiler.compile(ProjectionCompiler.java:393) > at > org.apache.phoenix.compile.QueryCompiler.compileSingleFlatQuery(QueryCompiler.java:537) > at > org.apache.phoenix.compile.QueryCompiler.compileSingleQuery(QueryCompiler.java:488) > at > org.apache.phoenix.compile.QueryCompiler.compileSelect(QueryCompiler.java:200) > at org.apache.phoenix.compile.QueryCompiler.compile(QueryCompiler.java:157) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:364) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:338) > at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:246) > at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250) > at sqlline.Commands.execute(Commands.java:822) > at sqlline.Commands.sql(Commands.java:732) > at sqlline.SqlLine.dispatch(SqlLine.java:808) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2074) StackOverflowError for hash join with round robin
[
https://issues.apache.org/jira/browse/PHOENIX-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613275#comment-14613275
]
Josh Mahonin commented on PHOENIX-2074:
---
I'm not sure if this is the same issue or not, but I'm running into something
similar with 4.4.0 with one of my spark jobs. This appears to be a regression
from 4.3.1. I don't yet have a lot of detail yet, but I am seeing the following
output from Spark:
java.lang.StackOverflowError
at
org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.next(RoundRobinResultIterator.java:309)
at
org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.next(RoundRobinResultIterator.java:309)
at
org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.next(RoundRobinResultIterator.java:309)
It's a bit difficult to update the cluster I'm seeing this on right now, so I
won't be able to test out this patch readily until I can get another
environment up and load the dataset in. I can say that there are no JOINs
involved, as this is going through the spark -> mapreduce pipeline. There's a
few SELECTs happening at the same time, but my guess is it's failing on the one
dataset which is quite a lot larger than the others at 6632640 rows.
The SELECT itself isn't complicated, I believe it's getting translated to
something like:
SELECT ETYPE,EID,M,E FROM WD WHERE TID = '???';
Where the DDL is approximately:
CREATE TABLE IF NOT EXISTS WD(
TID CHAR(3) NOT NULL,
ETYPE CHAR(3) NOT NULL,
EID VARCHAR NOT NULL,
M INTEGER NOT NULL,
E DOUBLE
CONSTRAINT pk PRIMARY KEY (TID, ETYPE, EID, M)) MULTI_TENANT=true,
COMPRESSION='SNAPPY', KEEP_DELETED_CELLS = false;
I don't see the same issues with the same job, on a similar dataset where the
231840 rows. Note that it's hitting the same table, just with a different TID.
I'll keep digging at it, and am willing and able to help out and test any
patches, this is pretty high priority for me to get working again.
> StackOverflowError for hash join with round robin
> -
>
> Key: PHOENIX-2074
> URL: https://issues.apache.org/jira/browse/PHOENIX-2074
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: PHOENIX-2074.patch
>
>
> EVENTS Table has id,article, and more columns. Id is the primay key
> MAPPING Table has id,article,category columns. Id is the primay key
> There is index on article column of both the tables.
> Below is the query.
> select count(MAPPING.article) as cnt,MAPPING.category from EVENTS
> join
> MAPPING on MAPPING.article = EVENTS.article
> group by category order by cnt ;
> Here's the stack trace:
> {code}
> Error: Encountered exception in sub plan [0] execution. (state=,code=0)
> java.sql.SQLException: Encountered exception in sub plan [0] execution.
> at
> org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:156)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:251)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: java.lang.StackOverflowError
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.close(RoundRobinResultIterator.java:298)
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.close(RoundRobinResultIterator.java:298)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2074) StackOverflowError for hash join with round robin
[
https://issues.apache.org/jira/browse/PHOENIX-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613436#comment-14613436
]
Josh Mahonin commented on PHOENIX-2074:
---
Well, this is a bit frustrating.
I regenerated my dataset, with the intent of creating differently-sized sets to
test where it falls over, and now I can't reproduce the problem. I've got an
artificially bumped up dataset at 11M rows, and everything is running just
fine. I haven't changed any other settings or config, and it's definitely still
using the RoundRobinIterator.
Is it possible HBase or Phoenix has done something, either with compactions, or
maybe something to do with metrics / guideposts, to change the previous
behaviour?
Will mull it over on the weekend.
> StackOverflowError for hash join with round robin
> -
>
> Key: PHOENIX-2074
> URL: https://issues.apache.org/jira/browse/PHOENIX-2074
> Project: Phoenix
> Issue Type: Bug
>Reporter: James Taylor
> Attachments: PHOENIX-2074.patch
>
>
> EVENTS Table has id,article, and more columns. Id is the primay key
> MAPPING Table has id,article,category columns. Id is the primay key
> There is index on article column of both the tables.
> Below is the query.
> select count(MAPPING.article) as cnt,MAPPING.category from EVENTS
> join
> MAPPING on MAPPING.article = EVENTS.article
> group by category order by cnt ;
> Here's the stack trace:
> {code}
> Error: Encountered exception in sub plan [0] execution. (state=,code=0)
> java.sql.SQLException: Encountered exception in sub plan [0] execution.
> at
> org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:156)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:251)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: java.lang.StackOverflowError
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.close(RoundRobinResultIterator.java:298)
> at
> org.apache.phoenix.iterate.RoundRobinResultIterator$RoundRobinIterator.close(RoundRobinResultIterator.java:298)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2036) PhoenixConfigurationUtil should provide a pre-normalize table name to PhoenixRuntime
[ https://issues.apache.org/jira/browse/PHOENIX-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614096#comment-14614096 ] Josh Mahonin commented on PHOENIX-2036: --- [~maghamraviki...@gmail.com] Sure thing. I think some of these tests were dependant on specific case-sensitivity behaviour, which may seems to have changed as a result of this patch.. I'll have a look either tomorrow or Monday. > PhoenixConfigurationUtil should provide a pre-normalize table name to > PhoenixRuntime > > > Key: PHOENIX-2036 > URL: https://issues.apache.org/jira/browse/PHOENIX-2036 > Project: Phoenix > Issue Type: Bug >Reporter: Siddhi Mehta >Assignee: maghamravikiran >Priority: Minor > Attachments: PHOENIX-2036-v1.patch, PHOENIX-2036-v1.patch, > PHOENIX-2036-v2.patch, PHOENIX-2036.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > I was trying a basic store using PhoenixHBaseStorage and ran into some issues > with it complaining about TableNotFoundException. > The table(CUSTOM_ENTITY."z02") in question exists. > Looking at the stacktrace I think its likely related to the change in > PHOENIX-1682 where phoenix runtime expects a pre-normalized table name. > We need to update > PhoenixConfigurationUtil.getSelectColumnMetadataList(Configuration) be pass a > pre-normalized table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2036) PhoenixConfigurationUtil should provide a pre-normalize table name to PhoenixRuntime
[ https://issues.apache.org/jira/browse/PHOENIX-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2036: -- Attachment: PHOENIX-2036-spark.patch Call SchemaUtil.getEscapedArgument before setting the input table name in PhoenixRDD. Although the only other invocations of this seem to be in integration tests, I feel like this is the right strategy. First, the unit tests don't have to change, so the existing API remains consistent. Secondly, I think it follows the principle of least astonishment. The tableName will be passed exactly as specified by the RDD/DataFrame, before getting escaped and passed to the the PhoenixRuntime. > PhoenixConfigurationUtil should provide a pre-normalize table name to > PhoenixRuntime > > > Key: PHOENIX-2036 > URL: https://issues.apache.org/jira/browse/PHOENIX-2036 > Project: Phoenix > Issue Type: Bug >Reporter: Siddhi Mehta >Assignee: maghamravikiran >Priority: Minor > Attachments: PHOENIX-2036-spark.patch, PHOENIX-2036-v1.patch, > PHOENIX-2036-v1.patch, PHOENIX-2036-v2.patch, PHOENIX-2036.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > I was trying a basic store using PhoenixHBaseStorage and ran into some issues > with it complaining about TableNotFoundException. > The table(CUSTOM_ENTITY."z02") in question exists. > Looking at the stacktrace I think its likely related to the change in > PHOENIX-1682 where phoenix runtime expects a pre-normalized table name. > We need to update > PhoenixConfigurationUtil.getSelectColumnMetadataList(Configuration) be pass a > pre-normalized table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2036) PhoenixConfigurationUtil should provide a pre-normalize table name to PhoenixRuntime
[ https://issues.apache.org/jira/browse/PHOENIX-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615043#comment-14615043 ] Josh Mahonin commented on PHOENIX-2036: --- [~maghamraviki...@gmail.com] Good point, I hadn't considered the fullTableName scenario. I'll rework this and add specific unit tests for that. By the way, some of this work overlaps with changes I made for PHOENIX-2088, do you see those changes going in, or are you looking at a different strategy there? > PhoenixConfigurationUtil should provide a pre-normalize table name to > PhoenixRuntime > > > Key: PHOENIX-2036 > URL: https://issues.apache.org/jira/browse/PHOENIX-2036 > Project: Phoenix > Issue Type: Bug >Reporter: Siddhi Mehta >Assignee: maghamravikiran >Priority: Minor > Attachments: PHOENIX-2036-spark.patch, PHOENIX-2036-v1.patch, > PHOENIX-2036-v1.patch, PHOENIX-2036-v2.patch, PHOENIX-2036.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > I was trying a basic store using PhoenixHBaseStorage and ran into some issues > with it complaining about TableNotFoundException. > The table(CUSTOM_ENTITY."z02") in question exists. > Looking at the stacktrace I think its likely related to the change in > PHOENIX-1682 where phoenix runtime expects a pre-normalized table name. > We need to update > PhoenixConfigurationUtil.getSelectColumnMetadataList(Configuration) be pass a > pre-normalized table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2036) PhoenixConfigurationUtil should provide a pre-normalize table name to PhoenixRuntime
[ https://issues.apache.org/jira/browse/PHOENIX-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2036: -- Attachment: PHOENIX-2036-spark-v2.patch Use Phoenix helper where possible for escaping in phoenix-spark Remove the 'buildSql' helpers and use the PhoenixConfigurationUtil / PhoenixConfiguration directly. Add a unit test to make sure we can load tables with schemas and escaped table names. NOTE: This may have merge conflicts with the patch in PHOENIX-2088. I can adjust this one, or that one, where necessary. > PhoenixConfigurationUtil should provide a pre-normalize table name to > PhoenixRuntime > > > Key: PHOENIX-2036 > URL: https://issues.apache.org/jira/browse/PHOENIX-2036 > Project: Phoenix > Issue Type: Bug >Reporter: Siddhi Mehta >Assignee: maghamravikiran >Priority: Minor > Attachments: PHOENIX-2036-spark-v2.patch, PHOENIX-2036-spark.patch, > PHOENIX-2036-v1.patch, PHOENIX-2036-v1.patch, PHOENIX-2036-v2.patch, > PHOENIX-2036.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > I was trying a basic store using PhoenixHBaseStorage and ran into some issues > with it complaining about TableNotFoundException. > The table(CUSTOM_ENTITY."z02") in question exists. > Looking at the stacktrace I think its likely related to the change in > PHOENIX-1682 where phoenix runtime expects a pre-normalized table name. > We need to update > PhoenixConfigurationUtil.getSelectColumnMetadataList(Configuration) be pass a > pre-normalized table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-1968: -- Fix Version/s: 4.4.1 4.5.0 5.0.0 > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.5.0, 4.4.1 > > Attachments: PHOENIX-1968.patch > > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-1968) Phoenix-Spark: Should support saving arrays
[ https://issues.apache.org/jira/browse/PHOENIX-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin resolved PHOENIX-1968. --- Resolution: Fixed > Phoenix-Spark: Should support saving arrays > --- > > Key: PHOENIX-1968 > URL: https://issues.apache.org/jira/browse/PHOENIX-1968 > Project: Phoenix > Issue Type: Bug >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 5.0.0, 4.5.0, 4.4.1 > > Attachments: PHOENIX-1968.patch > > > At present, we only use 'setObject' on the PreparedStatement when writing > back to Phoenix. This patch allows calling 'setArray' with the appropriate > sql base type when an array is passed in as a column value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2036) PhoenixConfigurationUtil should provide a pre-normalize table name to PhoenixRuntime
[ https://issues.apache.org/jira/browse/PHOENIX-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615946#comment-14615946 ] Josh Mahonin commented on PHOENIX-2036: --- Pushed to 4.x-*, 4.4-* and master > PhoenixConfigurationUtil should provide a pre-normalize table name to > PhoenixRuntime > > > Key: PHOENIX-2036 > URL: https://issues.apache.org/jira/browse/PHOENIX-2036 > Project: Phoenix > Issue Type: Bug >Reporter: Siddhi Mehta >Assignee: maghamravikiran >Priority: Minor > Attachments: PHOENIX-2036-spark-v2.patch, PHOENIX-2036-spark.patch, > PHOENIX-2036-v1.patch, PHOENIX-2036-v1.patch, PHOENIX-2036-v2.patch, > PHOENIX-2036.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > I was trying a basic store using PhoenixHBaseStorage and ran into some issues > with it complaining about TableNotFoundException. > The table(CUSTOM_ENTITY."z02") in question exists. > Looking at the stacktrace I think its likely related to the change in > PHOENIX-1682 where phoenix runtime expects a pre-normalized table name. > We need to update > PhoenixConfigurationUtil.getSelectColumnMetadataList(Configuration) be pass a > pre-normalized table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2088: -- Attachment: PHOENIX-2088-wip-v3.patch Old patch still applies cleanly, but this version removes come commented out code and a quick hack I meant to remove earlier from PhoenixRecordWritable. > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: maghamravikiran > Attachments: PHOENIX-2088-pig.patch, PHOENIX-2088-wip-v2.patch, > PHOENIX-2088-wip-v3.patch, PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2102) phoenix-pherf: Wrong parent pom version in 4.4-HBase-0.98
Josh Mahonin created PHOENIX-2102: - Summary: phoenix-pherf: Wrong parent pom version in 4.4-HBase-0.98 Key: PHOENIX-2102 URL: https://issues.apache.org/jira/browse/PHOENIX-2102 Project: Phoenix Issue Type: Bug Reporter: Josh Mahonin Assignee: Josh Mahonin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2102) phoenix-pherf: Wrong parent pom version in 4.4-HBase-0.98
[ https://issues.apache.org/jira/browse/PHOENIX-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2102: -- Attachment: PHOENIX-2102.patch For 4.4-HBase-0.98 branch > phoenix-pherf: Wrong parent pom version in 4.4-HBase-0.98 > - > > Key: PHOENIX-2102 > URL: https://issues.apache.org/jira/browse/PHOENIX-2102 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Fix For: 4.4.1 > > Attachments: PHOENIX-2102.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2025) Phoenix-core's hbase-default.xml prevents HBaseTestingUtility from starting up in client apps
[ https://issues.apache.org/jira/browse/PHOENIX-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617320#comment-14617320 ] Josh Mahonin commented on PHOENIX-2025: --- @tdsilva Sure thing > Phoenix-core's hbase-default.xml prevents HBaseTestingUtility from starting > up in client apps > - > > Key: PHOENIX-2025 > URL: https://issues.apache.org/jira/browse/PHOENIX-2025 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 5.0.0, 4.5.0, 4.4.1 > > Attachments: PHOENIX-2025-ClientPortIssue.patch, PHOENIX-2025.patch, > PHOENIX-2025_v2.patch > > > Phoenix seems to have long had its own version of hbase-default.xml as a test > resource in phoenix-core with a single setting to override > hbase.defaults.for.version.skip to true. Sometime around Phoenix 4.3, > phoenix-core seems to have been split into a main jar and a test jar, and the > hbase-default.xml went into the test jar. > The odd result of this is that in client apps that include the test jar, the > classloader in HBaseConfiguration.create() now sees Phoenix's > hbase-default.xml, rather than HBase's, and creates a Configuration object > without HBase's defaults. One major consequence of this is that the > HBaseTestingUtility can't start up, because it relies on those HBase defaults > being set. This is a huge problem in a client app that includes the > phoenix-core test jar in order to make use of the PhoenixTestDriver and > BaseTest classes; the upgrade to 4.3 breaks all tests using the > HBaseTestingUtility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (PHOENIX-2025) Phoenix-core's hbase-default.xml prevents HBaseTestingUtility from starting up in client apps
[ https://issues.apache.org/jira/browse/PHOENIX-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617320#comment-14617320 ] Josh Mahonin edited comment on PHOENIX-2025 at 7/7/15 8:30 PM: --- [~tdsilva] Sure thing was (Author: jmahonin): @tdsilva Sure thing > Phoenix-core's hbase-default.xml prevents HBaseTestingUtility from starting > up in client apps > - > > Key: PHOENIX-2025 > URL: https://issues.apache.org/jira/browse/PHOENIX-2025 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby > Fix For: 5.0.0, 4.5.0, 4.4.1 > > Attachments: PHOENIX-2025-ClientPortIssue.patch, PHOENIX-2025.patch, > PHOENIX-2025_v2.patch > > > Phoenix seems to have long had its own version of hbase-default.xml as a test > resource in phoenix-core with a single setting to override > hbase.defaults.for.version.skip to true. Sometime around Phoenix 4.3, > phoenix-core seems to have been split into a main jar and a test jar, and the > hbase-default.xml went into the test jar. > The odd result of this is that in client apps that include the test jar, the > classloader in HBaseConfiguration.create() now sees Phoenix's > hbase-default.xml, rather than HBase's, and creates a Configuration object > without HBase's defaults. One major consequence of this is that the > HBaseTestingUtility can't start up, because it relies on those HBase defaults > being set. This is a huge problem in a client app that includes the > phoenix-core test jar in order to make use of the PhoenixTestDriver and > BaseTest classes; the upgrade to 4.3 breaks all tests using the > HBaseTestingUtility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-2112) Phoenix-Spark need to support UTF8String for spark 1.4.0
[
https://issues.apache.org/jira/browse/PHOENIX-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Mahonin reassigned PHOENIX-2112:
-
Assignee: Josh Mahonin
> Phoenix-Spark need to support UTF8String for spark 1.4.0
>
>
> Key: PHOENIX-2112
> URL: https://issues.apache.org/jira/browse/PHOENIX-2112
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Yi Tian
>Assignee: Josh Mahonin
>
> In Spark 1.4.0, Phoenix-Spark will throw an exception when we put a filter
> like *a='a'* , because phoenix did not recognized {{UTF8String}} as a String
> Type, and then transform this expression to *a=a*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-2112) Phoenix-Spark need to support UTF8String for spark 1.4.0
[
https://issues.apache.org/jira/browse/PHOENIX-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Mahonin updated PHOENIX-2112:
--
Attachment: PHOENIX-2112-v2.patch
Update Yi Tian's patch with missing include statement. Also bump Spark version
to 1.4.0. These dependencies are 'provided' anyway, so this shouldn't affect
compatibility with previous versions of Spark.
No new unit tests are necessary here, the upgrade to 1.4.0 requires the
PhoenixRelation fix for all the tests to run.
Also clean up invoking the 'escapeSql' function to call the one provided in
phoenix.util.StringUtil instead
> Phoenix-Spark need to support UTF8String for spark 1.4.0
>
>
> Key: PHOENIX-2112
> URL: https://issues.apache.org/jira/browse/PHOENIX-2112
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Yi Tian
>Assignee: Josh Mahonin
> Attachments: PHOENIX-2112-v2.patch
>
>
> In Spark 1.4.0, Phoenix-Spark will throw an exception when we put a filter
> like *a='a'* , because phoenix did not recognized {{UTF8String}} as a String
> Type, and then transform this expression to *a=a*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2112) Phoenix-Spark need to support UTF8String for spark 1.4.0
[
https://issues.apache.org/jira/browse/PHOENIX-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625018#comment-14625018
]
Josh Mahonin commented on PHOENIX-2112:
---
[~jamestaylor] I believe it should be backwards compatible, although it looks
like they've moved that UTF8String class in 1.4. I'm still on 1.3 myself, so
I'll make sure that continues to work.
At some point we might have to look at setting up Maven profiles for different
spark versions, even if just for the integration tests, but I'm hoping to avoid
that for now.
> Phoenix-Spark need to support UTF8String for spark 1.4.0
>
>
> Key: PHOENIX-2112
> URL: https://issues.apache.org/jira/browse/PHOENIX-2112
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Yi Tian
>Assignee: Josh Mahonin
> Attachments: PHOENIX-2112-v2.patch
>
>
> In Spark 1.4.0, Phoenix-Spark will throw an exception when we put a filter
> like *a='a'* , because phoenix did not recognized {{UTF8String}} as a String
> Type, and then transform this expression to *a=a*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2112) Phoenix-Spark need to support UTF8String for spark 1.4.0
[
https://issues.apache.org/jira/browse/PHOENIX-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625258#comment-14625258
]
Josh Mahonin commented on PHOENIX-2112:
---
Can confirm with some quick testing that this version is backwards compatible
with Spark 1.3.
For posterity, my test plan was to try the following code in spark-shell with a
few scenarios:
{code}
import org.apache.phoenix.spark._
val df = sqlContext.load("org.apache.phoenix.spark", Map("table" ->
"SOME_TABLE", "zkUrl" -> "some_host"))
df.filter().count()
{code}
1) Run spark-shell for Spark 1.4 without patch. Throws ColumnNotFoundException,
which matches the bug report
2) Run spark-shell for Spark 1.4 with patch. Succeeds and returns expected count
3) Run spark-shell for Spark 1.3 without patch. Succeeds and returns expected
count
4) Run spark-shell for Spark 1.3 with patch. Succeeds and returns expected count
> Phoenix-Spark need to support UTF8String for spark 1.4.0
>
>
> Key: PHOENIX-2112
> URL: https://issues.apache.org/jira/browse/PHOENIX-2112
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Yi Tian
>Assignee: Josh Mahonin
> Attachments: PHOENIX-2112-v2.patch
>
>
> In Spark 1.4.0, Phoenix-Spark will throw an exception when we put a filter
> like *a='a'* , because phoenix did not recognized {{UTF8String}} as a String
> Type, and then transform this expression to *a=a*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2112) Phoenix-Spark need to support UTF8String for spark 1.4.0
[
https://issues.apache.org/jira/browse/PHOENIX-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625345#comment-14625345
]
Josh Mahonin commented on PHOENIX-2112:
---
Pushed to 4.4, 4.x and master branches.
> Phoenix-Spark need to support UTF8String for spark 1.4.0
>
>
> Key: PHOENIX-2112
> URL: https://issues.apache.org/jira/browse/PHOENIX-2112
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Yi Tian
>Assignee: Josh Mahonin
> Attachments: PHOENIX-2112-v2.patch
>
>
> In Spark 1.4.0, Phoenix-Spark will throw an exception when we put a filter
> like *a='a'* , because phoenix did not recognized {{UTF8String}} as a String
> Type, and then transform this expression to *a=a*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (PHOENIX-2112) Phoenix-Spark need to support UTF8String for spark 1.4.0
[
https://issues.apache.org/jira/browse/PHOENIX-2112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Mahonin resolved PHOENIX-2112.
---
Resolution: Fixed
Fix Version/s: 4.4.1
4.5.0
> Phoenix-Spark need to support UTF8String for spark 1.4.0
>
>
> Key: PHOENIX-2112
> URL: https://issues.apache.org/jira/browse/PHOENIX-2112
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.4.0
>Reporter: Yi Tian
>Assignee: Josh Mahonin
> Fix For: 4.5.0, 4.4.1
>
> Attachments: PHOENIX-2112-v2.patch
>
>
> In Spark 1.4.0, Phoenix-Spark will throw an exception when we put a filter
> like *a='a'* , because phoenix did not recognized {{UTF8String}} as a String
> Type, and then transform this expression to *a=a*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
Josh Mahonin created PHOENIX-2116: - Summary: phoenix-flume: Sink/Serializer should be extendable Key: PHOENIX-2116 URL: https://issues.apache.org/jira/browse/PHOENIX-2116 Project: Phoenix Issue Type: Improvement Affects Versions: 4.5.0, 4.4.1 Reporter: Josh Mahonin Assignee: Josh Mahonin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2116: -- Attachment: PHOENIX-2116.patch This patch allows setting a custom Flume serializer, rather than just the "regex" one in the EventSerializers enum. Includes an integration test, and examples of an empty sink and serializer. > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2116.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2116: -- Description: When using flume, often times custom serializers are necessary to transform data before sending to a sink. The existing Phoenix implementation however makes it difficult to extend and add new functionality. > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628409#comment-14628409 ] Josh Mahonin commented on PHOENIX-2116: --- a) Sure thing, I just followed the tests already there. Should I move others over as well (e.g., testConfiguration, testInvalidConfiguration, etc.)? b) Good idea, and I'd started that way, but I was having problems initializing them within the unit tests (newInstance() failed). I'll see what I can do here. > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634392#comment-14634392 ] Josh Mahonin commented on PHOENIX-2088: --- Totally agree on standardizing on a common interface. The spark integration tests fail with this patch, although on line 32 on ConfigurationUtil.scala if I replace 'setOutputTableName' with 'setColInfoTableName', that seems to get things going again. It's not clear to me if this supercedes the setOutputTableName, or if they're both necessary now? > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva > Attachments: PHOENIX-2088-4.4-HBase-0.98.patch, > PHOENIX-2088-pig.patch, PHOENIX-2088-wip-v2.patch, PHOENIX-2088-wip-v3.patch, > PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635789#comment-14635789 ] Josh Mahonin commented on PHOENIX-2088: --- [~tdsilva] The previous version of the patch (PHOENIX-2088-4.4-HBase-0.98.patch) had changes in 'DataFrameFunctions.scala' and 'ProductRDDFunctions.scala' to specifically instantiate a new Configuration and ColumnInfo metadata list on a per-partition basis, due to those classes not being serializable. If this patch is reasonably close to finished and is working for mapreduce / pig, I can see about working your new changes into the spark plugin, though I don't think I'll be able to get to it for a day or two. > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva > Attachments: PHOENIX-2088-4.4-HBase-0.98-v2.patch, > PHOENIX-2088-4.4-HBase-0.98.patch, PHOENIX-2088-pig.patch, > PHOENIX-2088-wip-v2.patch, PHOENIX-2088-wip-v3.patch, PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635815#comment-14635815 ] Josh Mahonin commented on PHOENIX-2088: --- [~jamestaylor] If [~tdsilva] is comfortable with making the changes, I have no problems reviewing it. It's a busy time for me right now, so unfortunately I don't think I'll get a chance to dive into this until some time tomorrow. The spark integration tests are a pretty good indicator if the changes are compatible or not. The trouble with Spark is it's attempt to serialize data to each worker, and it can do so in subtle and frustrating ways. If the 'ColumnInfoToStringEncoderDecoder' class is back, then the code in trunk should be pretty close to working I would think. However, if we need to derive the columns from the configuration object, then each partition needs to instantiate its own version (as per the previous patch) to prevent serialization. > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva > Attachments: PHOENIX-2088-4.4-HBase-0.98-v2.patch, > PHOENIX-2088-4.4-HBase-0.98.patch, PHOENIX-2088-pig.patch, > PHOENIX-2088-wip-v2.patch, PHOENIX-2088-wip-v3.patch, PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2088) Prevent splitting and recombining select expressions for MR integration
[ https://issues.apache.org/jira/browse/PHOENIX-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635952#comment-14635952 ] Josh Mahonin commented on PHOENIX-2088: --- +1. Good work [~tdsilva] > Prevent splitting and recombining select expressions for MR integration > --- > > Key: PHOENIX-2088 > URL: https://issues.apache.org/jira/browse/PHOENIX-2088 > Project: Phoenix > Issue Type: Bug >Reporter: James Taylor >Assignee: Thomas D'Silva > Attachments: PHOENIX-2088-4.4-HBase-0.98-v2.patch, > PHOENIX-2088-4.4-HBase-0.98-v3.patch, PHOENIX-2088-4.4-HBase-0.98.patch, > PHOENIX-2088-pig.patch, PHOENIX-2088-wip-v2.patch, PHOENIX-2088-wip-v3.patch, > PHOENIX-2088-wip.patch > > > We currently send in the select expressions for the MR integration with a > delimiter separated string, split based on the delimiter, and then recombine > again using a comma separator. This is problematic because the delimiter > character may appear in a select expression, thus breaking this logic. > Instead, we should use a comma as the delimiter and avoid splitting and > recombining as it's not necessary in that case. Instead, the entire string > can be used as-is in that case to form the select expressions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-2135) Use ColumnInfoToStringEncoderDecoder.encode/decode to store the column metadata in the configuration
[ https://issues.apache.org/jira/browse/PHOENIX-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin reassigned PHOENIX-2135: - Assignee: Josh Mahonin > Use ColumnInfoToStringEncoderDecoder.encode/decode to store the column > metadata in the configuration > > > Key: PHOENIX-2135 > URL: https://issues.apache.org/jira/browse/PHOENIX-2135 > Project: Phoenix > Issue Type: Improvement >Reporter: Thomas D'Silva >Assignee: Josh Mahonin > > In the saveToPhoenix method ProductRDDFunctions and DataFrameFunctions see if > we can serialize the column metadata to the configuration object using > ColumnInfoToStringEncoderDecoder.encode > and then in data.mapPartitions use ColumnInfoToStringEncoderDecoder.decode to > get the list of column infos. > Currently we call PhoenixConfigurationUtil.getUpsertColumnMetadataList for > each partition to get the column metadata. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1926) Attempt to update a record in HBase via Phoenix from a Spark job causes java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access
[ https://issues.apache.org/jira/browse/PHOENIX-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646000#comment-14646000 ] Josh Mahonin commented on PHOENIX-1926: --- Thanks for the updates [~dgoldenberg] What versions of Phoenix / HBase are you seeing the need for "hbase.defaults.for.version.skip=true" setting? > Attempt to update a record in HBase via Phoenix from a Spark job causes > java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > -- > > Key: PHOENIX-1926 > URL: https://issues.apache.org/jira/browse/PHOENIX-1926 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.3.1 > Environment: centos x86_64 GNU/Linux > Phoenix: 4.3.1 custom built for HBase 0.98.9, Hadoop 2.4.0 > HBase: 0.98.9-hadoop2 > Hadoop: 2.4.0 > Spark: spark-1.3.0-bin-hadoop2.4 >Reporter: Dmitry Goldenberg >Priority: Critical > > Performing an UPSERT from within a Spark job, > UPSERT INTO items(entry_id, prop1, pop2) VALUES(?, ?, ?) > causes > 15/04/26 18:22:06 WARN client.HTable: Error calling coprocessor service > org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for > row \x00\x00ITEMS > java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1007) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307) > at > org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:237) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:231) > at > org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:207) > at > org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:248) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:503) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:494) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:288) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:287) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:219) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:174) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:179) > ... > Caused by: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.
[jira] [Commented] (PHOENIX-2036) PhoenixConfigurationUtil should provide a pre-normalize table name to PhoenixRuntime
[ https://issues.apache.org/jira/browse/PHOENIX-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646002#comment-14646002 ] Josh Mahonin commented on PHOENIX-2036: --- Hi [~danmeany], if you could contribute a quick unit test for us, that'd be really helpful. Thanks! > PhoenixConfigurationUtil should provide a pre-normalize table name to > PhoenixRuntime > > > Key: PHOENIX-2036 > URL: https://issues.apache.org/jira/browse/PHOENIX-2036 > Project: Phoenix > Issue Type: Bug >Reporter: Siddhi Mehta >Assignee: maghamravikiran >Priority: Minor > Attachments: PHOENIX-2036-spark-v2.patch, PHOENIX-2036-spark.patch, > PHOENIX-2036-v1.patch, PHOENIX-2036-v1.patch, PHOENIX-2036-v2.patch, > PHOENIX-2036.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > I was trying a basic store using PhoenixHBaseStorage and ran into some issues > with it complaining about TableNotFoundException. > The table(CUSTOM_ENTITY."z02") in question exists. > Looking at the stacktrace I think its likely related to the change in > PHOENIX-1682 where phoenix runtime expects a pre-normalized table name. > We need to update > PhoenixConfigurationUtil.getSelectColumnMetadataList(Configuration) be pass a > pre-normalized table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-1926) Attempt to update a record in HBase via Phoenix from a Spark job causes java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access
[ https://issues.apache.org/jira/browse/PHOENIX-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646017#comment-14646017 ] Josh Mahonin commented on PHOENIX-1926: --- Interesting. We've been using the 4.3.1 release against HBase 0.98.12-hadoop2, with Spark 1.3.0 and haven't encountered that. I don't think we use a custom 'hbase-default.xml' file though, only an 'hbase-site.xml'. > Attempt to update a record in HBase via Phoenix from a Spark job causes > java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > -- > > Key: PHOENIX-1926 > URL: https://issues.apache.org/jira/browse/PHOENIX-1926 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.3.1 > Environment: centos x86_64 GNU/Linux > Phoenix: 4.3.1 custom built for HBase 0.98.9, Hadoop 2.4.0 > HBase: 0.98.9-hadoop2 > Hadoop: 2.4.0 > Spark: spark-1.3.0-bin-hadoop2.4 >Reporter: Dmitry Goldenberg >Priority: Critical > > Performing an UPSERT from within a Spark job, > UPSERT INTO items(entry_id, prop1, pop2) VALUES(?, ?, ?) > causes > 15/04/26 18:22:06 WARN client.HTable: Error calling coprocessor service > org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService for > row \x00\x00ITEMS > java.util.concurrent.ExecutionException: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620) > at > org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1007) > at > org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311) > at > org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307) > at > org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:237) > at > org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.(FromCompiler.java:231) > at > org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:207) > at > org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:248) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:503) > at > org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:494) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:295) > at > org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:288) > at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) > at > org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:287) > at > org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:219) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:174) > at > org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:179) > ... > Caused by: java.lang.IllegalAccessError: class > com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass > com.google.protobuf.LiteralByteString > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader
[jira] [Commented] (PHOENIX-2162) Exception trying to write an ARRAY of UNSIGNED_SMALLINT
[
https://issues.apache.org/jira/browse/PHOENIX-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658448#comment-14658448
]
Josh Mahonin commented on PHOENIX-2162:
---
Thanks for the report [~rcardin]
Changing the data types didn't work for you? Given the following DDL, what
happens if you run the Spark code below in spark-shell?
DDL
{code}
CREATE TABLE ARRAY_TEST_TABLE_SHORT (ID BIGINT NOT NULL PRIMARY KEY, SHORTARRAY
SMALLINT[]);
{code}
Spark
{code}
import org.apache.phoenix.spark._
val dataSet = List((1L, Array[java.lang.Short](1.toShort,2.toShort,3.toShort)))
sc.parallelize(dataSet).saveToPhoenix("ARRAY_TEST_TABLE_SHORT",
Seq("ID","SHORTARRAY"), zkUrl = Some("localhost"))
{code}
> Exception trying to write an ARRAY of UNSIGNED_SMALLINT
> ---
>
> Key: PHOENIX-2162
> URL: https://issues.apache.org/jira/browse/PHOENIX-2162
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.5.0
> Environment: - Windows 7
> - Spark 1.3.1
> - Scala 2.10.5
> - HBase 1.0.1.1
>Reporter: Riccardo Cardin
>
> I am using Phoenix version 4.5.0 and the phoenix-spark plugin to write into
> HBase an ARRAY of UNSIGNED_SMALLINT. As stated in the documentation, this
> type is mapped to the java type java.lang.Short.
> Using the saveToPhoenix method on a RDD and passing a Scala Array of Short I
> obtain the following stacktrace:
> {noformat}
> Caused by: java.lang.ClassCastException: [S cannot be cast to
> [Ljava.lang.Object;
> at
> org.apache.phoenix.schema.types.PUnsignedSmallintArray.isCoercibleTo(PUnsignedSmallintArray.java:81)
> at
> org.apache.phoenix.expression.LiteralExpression.newConstant(LiteralExpression.java:174)
> at
> org.apache.phoenix.expression.LiteralExpression.newConstant(LiteralExpression.java:157)
> at
> org.apache.phoenix.expression.LiteralExpression.newConstant(LiteralExpression.java:144)
> at
> org.apache.phoenix.compile.UpsertCompiler$UpsertValuesCompiler.visit(UpsertCompiler.java:872)
> at
> org.apache.phoenix.compile.UpsertCompiler$UpsertValuesCompiler.visit(UpsertCompiler.java:856)
> at org.apache.phoenix.parse.BindParseNode.accept(BindParseNode.java:47)
> at
> org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:745)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:550)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:538)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:318)
> at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:311)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:309)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:239)
> at
> org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:173)
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeBatch(PhoenixStatement.java:1315)
> {noformat}
> Changing the type of the column to CHAR(1) ARRAY and use an Array of String,
> the write operation succeds.
> I've tried to force to use an Array[java.lang.Short), to avoid mismatch
> between Scala and Java types, but I've obtained the same error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PHOENIX-2168) PUnsignedSmallIntArray is not coercible from short[]
Josh Mahonin created PHOENIX-2168: - Summary: PUnsignedSmallIntArray is not coercible from short[] Key: PHOENIX-2168 URL: https://issues.apache.org/jira/browse/PHOENIX-2168 Project: Phoenix Issue Type: Bug Affects Versions: 4.5.0 Reporter: Josh Mahonin Actually discovered from PHOENIX-2162, using 'connection.createArrayOf' for UNSIGNED_SMALLINT results in a stack trace, no matter if a Short[] or short[] is passed in (it seems to convert to primitives no matter what). Reproducible test case is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (PHOENIX-2168) PUnsignedSmallIntArray is not coercible from short[]
[ https://issues.apache.org/jira/browse/PHOENIX-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2168: -- Attachment: PHOENIX-2168-bug.patch > PUnsignedSmallIntArray is not coercible from short[] > > > Key: PHOENIX-2168 > URL: https://issues.apache.org/jira/browse/PHOENIX-2168 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.5.0 >Reporter: Josh Mahonin > Attachments: PHOENIX-2168-bug.patch > > > Actually discovered from PHOENIX-2162, using 'connection.createArrayOf' for > UNSIGNED_SMALLINT results in a stack trace, no matter if a Short[] or short[] > is passed in (it seems to convert to primitives no matter what). > Reproducible test case is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2168) PUnsignedSmallIntArray is not coercible from short[]
[ https://issues.apache.org/jira/browse/PHOENIX-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660101#comment-14660101 ] Josh Mahonin commented on PHOENIX-2168: --- Oops, dupe of PHOENIX-1967 > PUnsignedSmallIntArray is not coercible from short[] > > > Key: PHOENIX-2168 > URL: https://issues.apache.org/jira/browse/PHOENIX-2168 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.5.0 >Reporter: Josh Mahonin > Attachments: PHOENIX-2168-bug.patch > > > Actually discovered from PHOENIX-2162, using 'connection.createArrayOf' for > UNSIGNED_SMALLINT results in a stack trace, no matter if a Short[] or short[] > is passed in (it seems to convert to primitives no matter what). > Reproducible test case is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (PHOENIX-2168) PUnsignedSmallIntArray is not coercible from short[]
[ https://issues.apache.org/jira/browse/PHOENIX-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin resolved PHOENIX-2168. --- Resolution: Duplicate > PUnsignedSmallIntArray is not coercible from short[] > > > Key: PHOENIX-2168 > URL: https://issues.apache.org/jira/browse/PHOENIX-2168 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.5.0 >Reporter: Josh Mahonin > Attachments: PHOENIX-2168-bug.patch > > > Actually discovered from PHOENIX-2162, using 'connection.createArrayOf' for > UNSIGNED_SMALLINT results in a stack trace, no matter if a Short[] or short[] > is passed in (it seems to convert to primitives no matter what). > Reproducible test case is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (PHOENIX-2169) Illegal data error on UPSERT SELECT and JOIN with salted tables
Josh Mahonin created PHOENIX-2169:
-
Summary: Illegal data error on UPSERT SELECT and JOIN with salted
tables
Key: PHOENIX-2169
URL: https://issues.apache.org/jira/browse/PHOENIX-2169
Project: Phoenix
Issue Type: Bug
Affects Versions: 4.5.0
Reporter: Josh Mahonin
I have an issue where I get periodic failures (~50%) for an UPSERT SELECT query
involving a JOIN on salted tables. Unfortunately I haven't been able to create
a reproducible test case yet, though I'll keep trying. I believe this same
behaviour existed in 4.3.1 as well, so I don't think it's a regression.
The upsert query itself looks something like this:
{code}
UPSERT INTO a(tid, ds, etp, eid, ts, atp, rel, tp, tpid, dt, pro)
SELECT c.tid,
c.ds,
c.etp,
c.eid,
c.dh,
0,
c.rel,
c.tp,
c.tpid,
current_time(),
1.0 / s.th
FROM e_c c
join e_s s
ON s.tid = c.tid
ANDs.ds = c.ds
ANDs.etp = c.etp
ANDs.eid = c.eid
WHERE c.tid = 'FOO';
{code}
Without the upsert, the query always returns the right data, but with the
upsert, it ends up with failures like:
Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data.
Expected length of at least 109 bytes, but had 19 (state=22000,code=201)
The explain plan looks like:
{code}
UPSERT SELECT
CLIENT 16-CHUNK PARALLEL 16-WAY RANGE SCAN OVER E_C [0,'FOO']
SERVER FILTER BY FIRST KEY ONLY
PARALLEL INNER-JOIN TABLE 0
CLIENT 16-CHUNK PARALLEL 16-WAY FULL SCAN OVER E_S
DYNAMIC SERVER FILTER BY (C.TID, C.DS, C.ETP, C.EID) IN ((S.TID, S.DS,
S.ETP, S.EID))
{code}
I'm using SALT_BUCKETS=16 for both tables in the join, and this is a dev
environment, so only 1 region server. Note that without salted tables, I have
no issue with this query.
The number of rows in E_C is around 23K, and the number of rows in E_S is 62.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2169) Illegal data error on UPSERT SELECT and JOIN with salted tables
[
https://issues.apache.org/jira/browse/PHOENIX-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660439#comment-14660439
]
Josh Mahonin commented on PHOENIX-2169:
---
I should note that with DEBUG logging enabled, I can't see any discernible
difference between the successful and unsuccessful queries, though if there's
anything specific I should be looking for, please let me know.
> Illegal data error on UPSERT SELECT and JOIN with salted tables
> ---
>
> Key: PHOENIX-2169
> URL: https://issues.apache.org/jira/browse/PHOENIX-2169
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.5.0
>Reporter: Josh Mahonin
>
> I have an issue where I get periodic failures (~50%) for an UPSERT SELECT
> query involving a JOIN on salted tables. Unfortunately I haven't been able to
> create a reproducible test case yet, though I'll keep trying. I believe this
> same behaviour existed in 4.3.1 as well, so I don't think it's a regression.
> The upsert query itself looks something like this:
> {code}
> UPSERT INTO a(tid, ds, etp, eid, ts, atp, rel, tp, tpid, dt, pro)
> SELECT c.tid,
>c.ds,
>c.etp,
>c.eid,
>c.dh,
>0,
>c.rel,
>c.tp,
>c.tpid,
>current_time(),
>1.0 / s.th
> FROM e_c c
> join e_s s
> ON s.tid = c.tid
> ANDs.ds = c.ds
> ANDs.etp = c.etp
> ANDs.eid = c.eid
> WHERE c.tid = 'FOO';
> {code}
> Without the upsert, the query always returns the right data, but with the
> upsert, it ends up with failures like:
> Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data.
> Expected length of at least 109 bytes, but had 19 (state=22000,code=201)
> The explain plan looks like:
> {code}
> UPSERT SELECT
> CLIENT 16-CHUNK PARALLEL 16-WAY RANGE SCAN OVER E_C [0,'FOO']
> SERVER FILTER BY FIRST KEY ONLY
> PARALLEL INNER-JOIN TABLE 0
> CLIENT 16-CHUNK PARALLEL 16-WAY FULL SCAN OVER E_S
> DYNAMIC SERVER FILTER BY (C.TID, C.DS, C.ETP, C.EID) IN ((S.TID, S.DS,
> S.ETP, S.EID))
> {code}
> I'm using SALT_BUCKETS=16 for both tables in the join, and this is a dev
> environment, so only 1 region server. Note that without salted tables, I have
> no issue with this query.
> The number of rows in E_C is around 23K, and the number of rows in E_S is 62.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2169) Illegal data error on UPSERT SELECT and JOIN with salted tables
[
https://issues.apache.org/jira/browse/PHOENIX-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706068#comment-14706068
]
Josh Mahonin commented on PHOENIX-2169:
---
Including comments from [~johngouf] posted to phoenix-users:
Query:
{code}
UPSERT INTO READINGS
SELECT R.SMID, R.DT, R.US, R.GEN, R.USEST, R.GENEST, RM.LAT, RM.LON, RM.ZIP,
RM.FEEDER
FROM READINGS AS R
JOIN
(SELECT SMID,LAT,LON,ZIP,FEEDER
FROM READINGS_META) AS RM
ON R.SMID = RM.SMID
{code}
Stacktrace:
{noformat}
Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data.
Expected length of at least 70 bytes, but had 25 (state=22000,code=201)
java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000):
Illegal data. Expected length of at least 70 bytes, but had 25
at
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:388)
at
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
at
org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)
at
org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:553)
at
org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at
org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at
org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at
org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:685)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:314)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:306)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:304)
at
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1374)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
{noformat}
> Illegal data error on UPSERT SELECT and JOIN with salted tables
> ---
>
> Key: PHOENIX-2169
> URL: https://issues.apache.org/jira/browse/PHOENIX-2169
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.5.0
>Reporter: Josh Mahonin
>
> I have an issue where I get periodic failures (~50%) for an UPSERT SELECT
> query involving a JOIN on salted tables. Unfortunately I haven't been able to
> create a reproducible test case yet, though I'll keep trying. I believe this
> same behaviour existed in 4.3.1 as well, so I don't think it's a regression.
> The upsert query itself looks something like this:
> {code}
> UPSERT INTO a(tid, ds, etp, eid, ts, atp, rel, tp, tpid, dt, pro)
> SELECT c.tid,
>c.ds,
>c.etp,
>c.eid,
>c.dh,
>0,
>c.rel,
>c.tp,
>c.tpid,
>current_time(),
>1.0 / s.th
> FROM e_c c
> join e_s s
> ON s.tid = c.tid
> ANDs.ds = c.ds
> ANDs.etp = c.etp
> ANDs.eid = c.eid
> WHERE c.tid = 'FOO';
> {code}
> Without the upsert, the query always returns the right data, but with the
> upsert, it ends up with failures like:
> Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data.
> Expected length of at least 109 bytes, but had 19 (state=22000,code=201)
> The explain plan looks like:
> {code}
> UPSERT SELECT
> CLIENT 16-CHUNK PARALLEL 16-WAY RANGE SCAN OVER E_C [0,'FOO']
> SERVER FILTER BY FIRST KEY ONLY
> PARALLEL INNER-JOIN TABLE 0
> CLIENT 16-CHUNK PARALLEL 16-WAY FULL SCAN OVER E_S
> DYNAMIC SERVER FILTER BY (C.TID, C.DS, C.ETP, C.EID) IN ((S.TID, S.DS,
> S.ETP, S.EID))
> {code}
> I'm using SALT_BUCKETS=16 for both tables in the join, and this is a dev
> environment, so only 1 region server. Note that without salted tables, I have
> no issue with this query.
> The number of rows in E_C is around 23K, and the number of rows in E_S is 62.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (PHOENIX-2169) Illegal data error on UPSERT SELECT and JOIN with salted tables
[
https://issues.apache.org/jira/browse/PHOENIX-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Mahonin updated PHOENIX-2169:
--
Attachment: PHOENIX-2169-bug.patch
Test seems to invoke the problem. I think this might flap, but it fails pretty
consistently for me.
Removing the SALT_BUCKETS=4 the tables results in the test passing.
> Illegal data error on UPSERT SELECT and JOIN with salted tables
> ---
>
> Key: PHOENIX-2169
> URL: https://issues.apache.org/jira/browse/PHOENIX-2169
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.5.0
>Reporter: Josh Mahonin
> Attachments: PHOENIX-2169-bug.patch
>
>
> I have an issue where I get periodic failures (~50%) for an UPSERT SELECT
> query involving a JOIN on salted tables. Unfortunately I haven't been able to
> create a reproducible test case yet, though I'll keep trying. I believe this
> same behaviour existed in 4.3.1 as well, so I don't think it's a regression.
> The upsert query itself looks something like this:
> {code}
> UPSERT INTO a(tid, ds, etp, eid, ts, atp, rel, tp, tpid, dt, pro)
> SELECT c.tid,
>c.ds,
>c.etp,
>c.eid,
>c.dh,
>0,
>c.rel,
>c.tp,
>c.tpid,
>current_time(),
>1.0 / s.th
> FROM e_c c
> join e_s s
> ON s.tid = c.tid
> ANDs.ds = c.ds
> ANDs.etp = c.etp
> ANDs.eid = c.eid
> WHERE c.tid = 'FOO';
> {code}
> Without the upsert, the query always returns the right data, but with the
> upsert, it ends up with failures like:
> Error: ERROR 201 (22000): Illegal data. ERROR 201 (22000): Illegal data.
> Expected length of at least 109 bytes, but had 19 (state=22000,code=201)
> The explain plan looks like:
> {code}
> UPSERT SELECT
> CLIENT 16-CHUNK PARALLEL 16-WAY RANGE SCAN OVER E_C [0,'FOO']
> SERVER FILTER BY FIRST KEY ONLY
> PARALLEL INNER-JOIN TABLE 0
> CLIENT 16-CHUNK PARALLEL 16-WAY FULL SCAN OVER E_S
> DYNAMIC SERVER FILTER BY (C.TID, C.DS, C.ETP, C.EID) IN ((S.TID, S.DS,
> S.ETP, S.EID))
> {code}
> I'm using SALT_BUCKETS=16 for both tables in the join, and this is a dev
> environment, so only 1 region server. Note that without salted tables, I have
> no issue with this query.
> The number of rows in E_C is around 23K, and the number of rows in E_S is 62.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-2169) Illegal data error on UPSERT SELECT and JOIN with salted tables
[
https://issues.apache.org/jira/browse/PHOENIX-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706775#comment-14706775
]
Josh Mahonin commented on PHOENIX-2169:
---
This is the stack trace I get on the master branch:
{noformat}
org.apache.phoenix.exception.PhoenixIOException:
java.lang.ArrayIndexOutOfBoundsException
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:553)
at
org.apache.phoenix.iterate.MergeSortResultIterator.getIterators(MergeSortResultIterator.java:48)
at
org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:84)
at
org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:111)
at
org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at
org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:685)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:319)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:311)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:309)
at
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1432)
at
org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT.testUpsertSelectWithJoinOnSaltedTables(SaltedTableUpsertSelectIT.java:273)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: java.util.concurrent.ExecutionException:
java.lang.ArrayIndexOutOfBoundsException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:549)
... 40 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at
org.apache.phoenix.schema.types.PVarbinary.toObject(PVarbinary.java:79)
at
org.apache.phoenix.schema.types.PDataType.toObject(PDataType.java:984)
at
org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:75)
at
org.apache.phoenix.jdbc.PhoenixResultSet.getBytes(PhoenixResultSet.java:307)
at
org.apache.phoenix.compile.UpsertCompiler.upsertSelect(UpsertCompiler.java:139)
at
org.apache.phoenix.compile.UpsertCompiler.access$000(UpsertCompiler.java:98)
at
org.apache.phoenix.compile.UpsertCompiler$UpsertingParallelIteratorFactory.mutate(UpsertCompiler.java:
[jira] [Commented] (PHOENIX-2169) Illegal data error on UPSERT SELECT and JOIN with salted tables
[
https://issues.apache.org/jira/browse/PHOENIX-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706935#comment-14706935
]
Josh Mahonin commented on PHOENIX-2169:
---
I've reduced the for-loops down to 10 rows upserted per table, which still
usually results in a failure. Although I'm mostly seeing
ArrayIndexOutOfBoundsExceptions, I'm occasionally seeing this as well:
{noformat}
java.sql.SQLException: ERROR 218 (22018): Constraint violatioin. ERROR 218
(22018): Constraint violatioin. DEST.PK3 may not be null
at
org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:389)
at
org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
at
org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)
at
org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)
at
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:553)
at
org.apache.phoenix.iterate.MergeSortResultIterator.getIterators(MergeSortResultIterator.java:48)
at
org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:84)
at
org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:111)
at
org.apache.phoenix.iterate.DelegateResultIterator.next(DelegateResultIterator.java:44)
at
org.apache.phoenix.compile.UpsertCompiler$2.execute(UpsertCompiler.java:685)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:319)
at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:311)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:309)
at
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1432)
at
org.apache.phoenix.end2end.salted.SaltedTableUpsertSelectIT.testUpsertSelectWithJoinOnSaltedTables(SaltedTableUpsertSelectIT.java:273)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:78)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:212)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
{noformat}
> Illegal data error on UPSERT SELECT and JOIN with salted tables
> ---
>
> Key: PHOENIX-2169
> URL: https://issues.apache.org/jira/browse/PHOENIX-2169
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.5.0
>Reporter: Josh Mahonin
> Attachments: PHOENIX-2169-bug.patch
>
>
> I have an issue where I get periodic failures (~50%) for an UPSERT SELECT
> query involvi
[jira] [Updated] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin updated PHOENIX-2196: -- Attachment: PHOENIX-2196.patch > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Mahonin reassigned PHOENIX-2196: - Assignee: Josh Mahonin > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709622#comment-14709622 ] Josh Mahonin commented on PHOENIX-2196: --- Hi [~randerzander] I just uploaded a modified patch based on your PR. The only main difference is we're invoking 'SchemaUtil.normalizeIdentifier()', which under the hood does the same transformation logic you had anyway. It also includes a unit test that works on one of the existing test tables with both upper and lowercase table names. I tried to use the spark-csv library for the unit tests as a complete example, but a really useful helper method they added recently, csvRdd(), isn't available in the released version. If you could test this patch out on your end and let us know if it works for you, that'd be great. > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2196) phoenix-spark should automatically convert DataFrame field names to all caps
[ https://issues.apache.org/jira/browse/PHOENIX-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14709627#comment-14709627 ] Josh Mahonin commented on PHOENIX-2196: --- I just realized I created the diff off of [~randerzander]'s PR, so this won't apply cleanly to master. I'll clean it up once we hear back from him. > phoenix-spark should automatically convert DataFrame field names to all caps > > > Key: PHOENIX-2196 > URL: https://issues.apache.org/jira/browse/PHOENIX-2196 > Project: Phoenix > Issue Type: Improvement >Reporter: Randy Gelhausen >Assignee: Josh Mahonin >Priority: Minor > Attachments: PHOENIX-2196.patch > > > phoenix-spark will fail to save a DF into a Phoenix table if the DataFrame's > fields are not all capitalized. Since Phoenix internally capitalizes all > column names, the DataFrame.save method should automatically capitalize DF > field names as a convenience to the end user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-2116) phoenix-flume: Sink/Serializer should be extendable
[ https://issues.apache.org/jira/browse/PHOENIX-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710069#comment-14710069 ] Josh Mahonin commented on PHOENIX-2116: --- Have updated the patch. Only notable changes are in the unit tests from previous version: Comments: a) Unfortunately, the act of calling 'configure()' on PhoenixSink seems to require a running cluster. b) I've created a test which calls a mock instance of the PhoenixSink, to verify that configure() is invoked. However, the serializer is a bit trickier, since it's instantiated using a Class.forName(). Instead, I've made a CustomSerializer which writes some data to Phoenix, then the test verifies the data was written. > phoenix-flume: Sink/Serializer should be extendable > --- > > Key: PHOENIX-2116 > URL: https://issues.apache.org/jira/browse/PHOENIX-2116 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 4.5.0, 4.4.1 >Reporter: Josh Mahonin >Assignee: Josh Mahonin > Attachments: PHOENIX-2116-v2.patch, PHOENIX-2116.patch > > > When using flume, often times custom serializers are necessary to transform > data before sending to a sink. The existing Phoenix implementation however > makes it difficult to extend and add new functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
