For example 1 such error in QueryExecutionServiceImp.java Lens Server
project
private LensResultSet getResultset(QueryHandle queryHandle) throws
LensException {
QueryContext ctx = allQueries.get(queryHandle);
if (ctx == null) {
return getResultsetFromDAO(queryHandle);
} else {
synchronized (ctx) {
if (ctx.isFinishedQueryPersisted()) {
return getResultsetFromDAO(queryHandle);
}
LensResultSet resultSet = resultSets.get(queryHandle);
if (resultSet == null) {
if (ctx.isPersistent() && ctx.getQueryOutputFormatter() != null) {
resultSets
.put(queryHandle,
new LensPersistentResult(
ctx.getQueryOutputFormatter().getMetadata(),
ctx.getQueryOutputFormatter().getFinalOutputPath(),
ctx.getQueryOutputFormatter().getNumRows()));
} else if (allQueries.get(queryHandle).isResultAvailableInDriver()) {
resultSet =
allQueries.get(queryHandle).getSelectedDriver().fetchResultSet(allQueries.get(queryHandle));
resultSets.put(queryHandle, resultSet);
} else {
throw new NotFoundException("Result set not available for
query:" + queryHandle);
}
}
}
if (resultSets.get(queryHandle) instanceof InMemoryResultSet) {
((InMemoryResultSet) resultSets.get(queryHandle)).seekToStart();
}
return resultSets.get(queryHandle);
}
}
There are no such methods in QueryContext() but annotated with @Getter
and @Setter. Do we need any plugin to solve it?
Thanks
Ajay
On Fri, Jul 24, 2015 at 2:33 PM, Rajat Khandelwal <[email protected]>
wrote:
> Yes, can you paste the exact error?
>
> On Fri, Jul 24, 2015 at 2:11 PM Yash Sharma <[email protected]> wrote:
>
> > Its certainly something wrt Env. Can you just copy paste couple of
> errors ?
> >
> > On Fri, Jul 24, 2015 at 2:00 PM, Ajay <[email protected]> wrote:
> >
> > > I did the same from command line.
> > >
> > > mvn clean install -DskipTests -DskipCheck
> > > mvn idea:idea
> > >
> > >
> > > Java version is 1.8
> > >
> > > I tried opening the project directly and selected
> > > /home/ajay/openSource/incubator-lens. Also I tried importing as maven
> > > project. Both the cases I see compilation errors.
> > >
> > > -
> > > Ajay
> > >
> > >
> > > On Fri, Jul 24, 2015 at 1:42 PM, Rajat Khandelwal <
> > [email protected]>
> > > wrote:
> > >
> > > > I'm using idea too. There are no compilation issues. Can you do
> > > >
> > > > mvn clean install -DskipTests -DskipCheck
> > > > mvn idea:idea
> > > >
> > > > from command line.
> > > >
> > > >
> > > > Also, check your java version. Lens is configured for java >=7
> > > >
> > > > On Fri, Jul 24, 2015 at 1:36 PM Ajay <[email protected]> wrote:
> > > >
> > > > > From console, the build goes thru fine.
> > > > >
> > > > > From Idea IDE, it is giving compilation errors (say one such file
> > > > > QueryExecutionServiceImp.java
> > > > > Lens Server project).
> > > > >
> > > > > Are you guys using Eclipse?
> > > > >
> > > > >
> > > > > On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <[email protected]>
> > > wrote:
> > > > >
> > > > > > Could you share the error. Might be env issues.
> > > > > > Also can you build from the console directly ?
> > > > > >
> > > > > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <[email protected]>
> > wrote:
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > I am using intelliJ Idea IDE. I checked out the source from
> GIT.
> > > But
> > > > I
> > > > > am
> > > > > > > seeing some compilation issues. Is there any documentation or
> > notes
> > > > on
> > > > > > how
> > > > > > > to set up the development envirnment?
> > > > > > >
> > > > > > > -
> > > > > > > Ajay
> > > > > > >
> > > > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <
> > > > > [email protected]
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Ajay,
> > > > > > > >
> > > > > > > > Nice to see your interest in exploring Lens.
> > > > > > > >
> > > > > > > > Looking at the requirement posted by you, seems you are only
> > > > > interested
> > > > > > > in
> > > > > > > > REST api on top of cassandra. See the lens REST api for query
> > > > service
> > > > > > > here
> > > > > > > > -
> > > > >
> http://lens.incubator.apache.org/resource_QueryServiceResource.html
> > > > > > > >
> > > > > > > > Lens provides REST api for query actions (submit, fetch
> > results,
> > > > > > cancel,
> > > > > > > > getstatus), query history and OLAP data model.
> > > > > > > >
> > > > > > > > If you are not interested in OLAP data model, you can simple
> > run
> > > > > native
> > > > > > > > queries on lens which would give REST api and history for you
> > as
> > > > > well.
> > > > > > > For
> > > > > > > > more details on OLAP data model, you can read the doc on
> > website.
> > > > > > > >
> > > > > > > > For new driver to be rewritten, it need not know anything
> about
> > > > OLAP
> > > > > > data
> > > > > > > > model. But all drivers are given HiveQL query to execute, if
> > > driver
> > > > > > > accepts
> > > > > > > > Hive QL directly you dont have to do much on the query
> itself.
> > If
> > > > > not,
> > > > > > > the
> > > > > > > > driver has to translate the query into something driver can
> > > > > understand.
> > > > > > > >
> > > > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can
> have
> > > > look
> > > > > at
> > > > > > > > JDBCDriver for converting into SQL. Hive QL is almost SQL,
> > except
> > > > it
> > > > > > > might
> > > > > > > > have to map some UDFs to SQL udfs.
> > > > > > > >
> > > > > > > > ok. Coming back to your questions, answers are inline.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <[email protected]>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > Couple of questions (that comes to my mind right now)
> > > > > > > > >
> > > > > > > > > 1) To create a Cluster instance to connect to a Cassandra
> > > cluster
> > > > > > > > > (Cassandra Java driver), we need to know the following:
> > > > > > > > > a) ip address and port no of few instances of the cluster
> > > > > > > > > b) authentication (if enabled)
> > > > > > > > > c) few more configuration (loading balancing policy, ret
> > policy
> > > > and
> > > > > > > > > compression etc..)
> > > > > > > > >
> > > > > > > > > Lens drivers are following xml based configuration for
> > passing
> > > > > > > > information. See
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml
> > > > > > > > and
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml
> > > > > > > > ,
> > > > > > > > how driver specific configuration is put. Also read
> > > > > > > > http://lens.incubator.apache.org/admin/config-server.html to
> > > > > > understand
> > > > > > > > configuration in lens.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Do we have any configuration support(yaml file or any
> other)
> > in
> > > > > Lens.
> > > > > > > If
> > > > > > > > > yes, any pointers (java file names or WIKI)
> > > > > > > > >
> > > > > > > > > 2) Cassandra cluser/session and prepared statements are
> multi
> > > > > thread
> > > > > > > safe
> > > > > > > > > and recommended to be cached and reused.
> > > > > > > > >
> > > > > > > > > How does Lens support Caching (in memory or distributed)?.
> > Any
> > > > > > pointers
> > > > > > > > > (java file names or WIKI)
> > > > > > > > >
> > > > > > > > > I did not understand the caching part here. But if you are
> > > > talking
> > > > > > > about
> > > > > > > > cassandra sessions to be used across user queries, you can
> look
> > > at
> > > > > how
> > > > > > > hive
> > > > > > > > sessions and connections are used in HiveDriver -
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <
> > > [email protected]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > That is great. We can probably pick the most recent Java
> > > driver
> > > > > and
> > > > > > > > > > Cassandra version then. Since it addresses couple of old
> > > > issues.
> > > > > > > > > >
> > > > > > > > > > Regarding Spark SQL for querying Cassandra I would let
> > other
> > > > > > > > contributors
> > > > > > > > > > suggest.
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <
> > [email protected]
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Yash.
> > > > > > > > > > >
> > > > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra
> > Java
> > > > > driver
> > > > > > > > [1].
> > > > > > > > > > What
> > > > > > > > > > > is the Cassandra and Java driver version we plan to
> > > > support?. I
> > > > > > > have
> > > > > > > > > > worked
> > > > > > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and
> > > ALLOW
> > > > > > > > FILTERING
> > > > > > > > > > > worked.
> > > > > > > > > > >
> > > > > > > > > > > Secondly, as you mentioned I am aware of these
> limitation
> > > in
> > > > > CQL.
> > > > > > > But
> > > > > > > > > > more
> > > > > > > > > > > features recently added in Cassandra 2.2.x. Also, other
> > > > option
> > > > > to
> > > > > > > > work
> > > > > > > > > > > around this to use Cassandra Spark connector and use
> > Spark
> > > > SQL.
> > > > > > > > > > >
> > > > > > > > > > > 1)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <
> > > > > [email protected]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Just to add on...
> > > > > > > > > > > >
> > > > > > > > > > > > There will be few challenges to support entire SQL on
> > > > > Cassandra
> > > > > > > > > > however.
> > > > > > > > > > > > Last time I tried translating SQL to CQL I faced
> couple
> > > of
> > > > > > > > scenarios
> > > > > > > > > -
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Filtering non indexed column in Cassandra
> > > > > > > > > > > > 2. Filtering by subset of primary key
> > > > > > > > > > > > 3. OR condition in where clause
> > > > > > > > > > > >
> > > > > > > > > > > > You can probably start without these being a blocker
> -
> > In
> > > > > > > parallel
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > discuss how this can be implemented on our stack.
> > Others
> > > > can
> > > > > > > > provide
> > > > > > > > > > > their
> > > > > > > > > > > > opinions here.
> > > > > > > > > > > >
> > > > > > > > > > > > Hope its helpful.
> > > > > > > > > > > >
> > > > > > > > > > > > -----
> > > > > > > > > > > > Examples:
> > > > > > > > > > > > Here 'trending_now' is a dummy table with (id, rank,
> > > > pog_id)
> > > > > > > where
> > > > > > > > > > > > (id,rank) is primary key pair.
> > > > > > > > > > > >
> > > > > > > > > > > > 1.
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > > > pog_id=10004 ;
> > > > > > > > > > > > Bad Request: No indexed columns present in by-columns
> > > > clause
> > > > > > with
> > > > > > > > > Equal
> > > > > > > > > > > > operator
> > > > > > > > > > > >
> > > > > > > > > > > > 2.
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> rank=4;
> > > > > > > > > > > > Bad Request: Cannot execute this query as it might
> > > involve
> > > > > data
> > > > > > > > > > filtering
> > > > > > > > > > > > and thus may have unpredictable performance. If you
> > want
> > > to
> > > > > > > execute
> > > > > > > > > > this
> > > > > > > > > > > > query despite the performance unpredictability, use
> > ALLOW
> > > > > > > FILTERING
> > > > > > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra
> java
> > > > > driver
> > > > > > as
> > > > > > > > of
> > > > > > > > > > now.
> > > > > > > > > > > >
> > > > > > > > > > > > 3.
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where rank=4
> > or
> > > > > > > > id='id0004';
> > > > > > > > > > > > Bad Request: line 1:40 missing EOF at 'or'
> > > > > > > > > > > >
> > > > > > > > > > > > 4. Valid Query:
> > > > > > > > > > > > cqlsh:recsys> select * from trending_now where
> > > id='id0004'
> > > > > and
> > > > > > > > > rank=4;
> > > > > > > > > > > >
> > > > > > > > > > > > id | rank | pog_id
> > > > > > > > > > > > --------+------+--------
> > > > > > > > > > > > id0004 | 4 | 10002
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma <
> > > > > > [email protected]>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The basic idea is to translate the Lens Query plan
> > > (Which
> > > > > is
> > > > > > a
> > > > > > > > Hive
> > > > > > > > > > > AST)
> > > > > > > > > > > > > to the DataStore specific Plan/Query.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The best example I can think of right now is the
> > > Elastic
> > > > > > Search
> > > > > > > > > patch
> > > > > > > > > > > by
> > > > > > > > > > > > > Amruth. You can go through the JIRA [1] for the
> > > detailed
> > > > > > > > discussion
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > Review Board [2] for code reference.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252
> > > > > > > > > > > > > 2. https://reviews.apache.org/r/36434
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <
> > > > > [email protected]>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Thanks Yash.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Is there any documentation or WIKI on the Lens
> > > Driver. I
> > > > > am
> > > > > > > > going
> > > > > > > > > > > > through
> > > > > > > > > > > > >> the code as well.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks
> > > > > > > > > > > > >> Ajay
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma <
> > > > > > > > [email protected]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi Ajay,
> > > > > > > > > > > > >> > Welcome to the Lens Dev.
> > > > > > > > > > > > >> > We do have plans for a Cassandra driver [1] for
> > Lens
> > > > but
> > > > > > no
> > > > > > > > one
> > > > > > > > > > has
> > > > > > > > > > > > >> picked
> > > > > > > > > > > > >> > it up yet. It would be great if you can pick up
> > the
> > > > task
> > > > > > and
> > > > > > > > > > submit
> > > > > > > > > > > a
> > > > > > > > > > > > >> patch
> > > > > > > > > > > > >> > for review.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Also drop a note on the list in case you stumble
> > > upon
> > > > > any
> > > > > > > > issue.
> > > > > > > > > > > > Someone
> > > > > > > > > > > > >> > will always be around to help you out.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > 1.
> https://issues.apache.org/jira/browse/LENS-654
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay <
> > > > > > > [email protected]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Hi,
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > I recently noticed about Apache Lens project.
> > > > > Currently
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > >> building
> > > > > > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary)
> as
> > > > there
> > > > > no
> > > > > > > > such
> > > > > > > > > > > Apache
> > > > > > > > > > > > >> open
> > > > > > > > > > > > >> > > source project exists. Now as Lens supports
> REST
> > > > APIs
> > > > > > for
> > > > > > > > > Hadoop
> > > > > > > > > > > and
> > > > > > > > > > > > >> > JDBC,
> > > > > > > > > > > > >> > > I want to know is there any plan to support
> for
> > > > Apache
> > > > > > > > > Cassandra
> > > > > > > > > > > as
> > > > > > > > > > > > >> its
> > > > > > > > > > > > >> > > support CQL and Spark SQL (thru Spark
> connector)
> > > > which
> > > > > > are
> > > > > > > > > more
> > > > > > > > > > > SQL
> > > > > > > > > > > > >> like.
> > > > > > > > > > > > >> > > If yes, I wish to know the details and
> > contribute
> > > as
> > > > > > well.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Thanks
> > > > > > > > > > > > >> > > Ajay
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
