>From console, the build goes thru fine. >From Idea IDE, it is giving compilation errors (say one such file QueryExecutionServiceImp.java Lens Server project).
Are you guys using Eclipse? On Fri, Jul 24, 2015 at 1:14 PM, Yash Sharma <[email protected]> wrote: > Could you share the error. Might be env issues. > Also can you build from the console directly ? > > On Fri, Jul 24, 2015 at 12:54 PM, Ajay <[email protected]> wrote: > > > Thanks. > > > > I am using intelliJ Idea IDE. I checked out the source from GIT. But I am > > seeing some compilation issues. Is there any documentation or notes on > how > > to set up the development envirnment? > > > > - > > Ajay > > > > On Fri, Jul 24, 2015 at 11:44 AM, amareshwarisr . <[email protected] > > > > wrote: > > > > > Hello Ajay, > > > > > > Nice to see your interest in exploring Lens. > > > > > > Looking at the requirement posted by you, seems you are only interested > > in > > > REST api on top of cassandra. See the lens REST api for query service > > here > > > - http://lens.incubator.apache.org/resource_QueryServiceResource.html > > > > > > Lens provides REST api for query actions (submit, fetch results, > cancel, > > > getstatus), query history and OLAP data model. > > > > > > If you are not interested in OLAP data model, you can simple run native > > > queries on lens which would give REST api and history for you as well. > > For > > > more details on OLAP data model, you can read the doc on website. > > > > > > For new driver to be rewritten, it need not know anything about OLAP > data > > > model. But all drivers are given HiveQL query to execute, if driver > > accepts > > > Hive QL directly you dont have to do much on the query itself. If not, > > the > > > driver has to translate the query into something driver can understand. > > > > > > For example : JDBCDriver converts HiveQL to SQL. - You can have look at > > > JDBCDriver for converting into SQL. Hive QL is almost SQL, except it > > might > > > have to map some UDFs to SQL udfs. > > > > > > ok. Coming back to your questions, answers are inline. > > > > > > > > > On Fri, Jul 24, 2015 at 9:08 AM, Ajay <[email protected]> wrote: > > > > > > > Thanks. > > > > > > > > Couple of questions (that comes to my mind right now) > > > > > > > > 1) To create a Cluster instance to connect to a Cassandra cluster > > > > (Cassandra Java driver), we need to know the following: > > > > a) ip address and port no of few instances of the cluster > > > > b) authentication (if enabled) > > > > c) few more configuration (loading balancing policy, ret policy and > > > > compression etc..) > > > > > > > > Lens drivers are following xml based configuration for passing > > > information. See > > > > > > > > > https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/resources/hivedriver-default.xml > > > and > > > > > > > > > https://github.com/apache/incubator-lens/blob/master/lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml > > > , > > > how driver specific configuration is put. Also read > > > http://lens.incubator.apache.org/admin/config-server.html to > understand > > > configuration in lens. > > > > > > > > > > Do we have any configuration support(yaml file or any other) in Lens. > > If > > > > yes, any pointers (java file names or WIKI) > > > > > > > > 2) Cassandra cluser/session and prepared statements are multi thread > > safe > > > > and recommended to be cached and reused. > > > > > > > > How does Lens support Caching (in memory or distributed)?. Any > pointers > > > > (java file names or WIKI) > > > > > > > > I did not understand the caching part here. But if you are talking > > about > > > cassandra sessions to be used across user queries, you can look at how > > hive > > > sessions and connections are used in HiveDriver - > > > > > > > > > https://github.com/apache/incubator-lens/blob/master/lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2015 at 11:16 PM, Yash Sharma <[email protected]> > > wrote: > > > > > > > > > That is great. We can probably pick the most recent Java driver and > > > > > Cassandra version then. Since it addresses couple of old issues. > > > > > > > > > > Regarding Spark SQL for querying Cassandra I would let other > > > contributors > > > > > suggest. > > > > > > > > > > On Thu, Jul 23, 2015 at 11:04 PM, Ajay <[email protected]> > wrote: > > > > > > > > > > > Thanks Yash. > > > > > > > > > > > > In Between, ALLOW FILTERING is supported in Cassandra Java driver > > > [1]. > > > > > What > > > > > > is the Cassandra and Java driver version we plan to support?. I > > have > > > > > worked > > > > > > on Cassandra 2.0.x and 2.1.x and Java driver 2.1.x and ALLOW > > > FILTERING > > > > > > worked. > > > > > > > > > > > > Secondly, as you mentioned I am aware of these limitation in CQL. > > But > > > > > more > > > > > > features recently added in Cassandra 2.2.x. Also, other option to > > > work > > > > > > around this to use Cassandra Spark connector and use Spark SQL. > > > > > > > > > > > > 1) > > > > > > > > > > > > > > > > > > > > > > > > > > > http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/querybuilder/Select.html > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2015 at 10:07 PM, Yash Sharma <[email protected] > > > > > > wrote: > > > > > > > > > > > > > Just to add on... > > > > > > > > > > > > > > There will be few challenges to support entire SQL on Cassandra > > > > > however. > > > > > > > Last time I tried translating SQL to CQL I faced couple of > > > scenarios > > > > - > > > > > > > > > > > > > > 1. Filtering non indexed column in Cassandra > > > > > > > 2. Filtering by subset of primary key > > > > > > > 3. OR condition in where clause > > > > > > > > > > > > > > You can probably start without these being a blocker - In > > parallel > > > we > > > > > can > > > > > > > discuss how this can be implemented on our stack. Others can > > > provide > > > > > > their > > > > > > > opinions here. > > > > > > > > > > > > > > Hope its helpful. > > > > > > > > > > > > > > ----- > > > > > > > Examples: > > > > > > > Here 'trending_now' is a dummy table with (id, rank, pog_id) > > where > > > > > > > (id,rank) is primary key pair. > > > > > > > > > > > > > > 1. > > > > > > > cqlsh:recsys> select * from trending_now where pog_id=10004 ; > > > > > > > Bad Request: No indexed columns present in by-columns clause > with > > > > Equal > > > > > > > operator > > > > > > > > > > > > > > 2. > > > > > > > cqlsh:recsys> select * from trending_now where rank=4; > > > > > > > Bad Request: Cannot execute this query as it might involve data > > > > > filtering > > > > > > > and thus may have unpredictable performance. If you want to > > execute > > > > > this > > > > > > > query despite the performance unpredictability, use ALLOW > > FILTERING > > > > > > > P.S. ALLOW FILTERING is not permitted in Cassandra java driver > as > > > of > > > > > now. > > > > > > > > > > > > > > 3. > > > > > > > cqlsh:recsys> select * from trending_now where rank=4 or > > > id='id0004'; > > > > > > > Bad Request: line 1:40 missing EOF at 'or' > > > > > > > > > > > > > > 4. Valid Query: > > > > > > > cqlsh:recsys> select * from trending_now where id='id0004' and > > > > rank=4; > > > > > > > > > > > > > > id | rank | pog_id > > > > > > > --------+------+-------- > > > > > > > id0004 | 4 | 10002 > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:45 PM, Yash Sharma < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > The basic idea is to translate the Lens Query plan (Which is > a > > > Hive > > > > > > AST) > > > > > > > > to the DataStore specific Plan/Query. > > > > > > > > > > > > > > > > The best example I can think of right now is the Elastic > Search > > > > patch > > > > > > by > > > > > > > > Amruth. You can go through the JIRA [1] for the detailed > > > discussion > > > > > and > > > > > > > the > > > > > > > > Review Board [2] for code reference. > > > > > > > > > > > > > > > > Best regards > > > > > > > > > > > > > > > > 1. https://issues.apache.org/jira/browse/LENS-252 > > > > > > > > 2. https://reviews.apache.org/r/36434 > > > > > > > > > > > > > > > > On Thu, Jul 23, 2015 at 9:30 PM, Ajay <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > >> Thanks Yash. > > > > > > > >> > > > > > > > >> Is there any documentation or WIKI on the Lens Driver. I am > > > going > > > > > > > through > > > > > > > >> the code as well. > > > > > > > >> > > > > > > > >> Thanks > > > > > > > >> Ajay > > > > > > > >> > > > > > > > >> > > > > > > > >> On Thu, Jul 23, 2015 at 12:01 PM, Yash Sharma < > > > [email protected]> > > > > > > > wrote: > > > > > > > >> > > > > > > > >> > Hi Ajay, > > > > > > > >> > Welcome to the Lens Dev. > > > > > > > >> > We do have plans for a Cassandra driver [1] for Lens but > no > > > one > > > > > has > > > > > > > >> picked > > > > > > > >> > it up yet. It would be great if you can pick up the task > and > > > > > submit > > > > > > a > > > > > > > >> patch > > > > > > > >> > for review. > > > > > > > >> > > > > > > > > >> > Also drop a note on the list in case you stumble upon any > > > issue. > > > > > > > Someone > > > > > > > >> > will always be around to help you out. > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > 1. https://issues.apache.org/jira/browse/LENS-654 > > > > > > > >> > > > > > > > > >> > On Thu, Jul 23, 2015 at 11:41 AM, Ajay < > > [email protected]> > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > Hi, > > > > > > > >> > > > > > > > > > >> > > I recently noticed about Apache Lens project. Currently > we > > > are > > > > > > > >> building > > > > > > > >> > > REST APIs for Apache Cassandra (proprietary) as there no > > > such > > > > > > Apache > > > > > > > >> open > > > > > > > >> > > source project exists. Now as Lens supports REST APIs > for > > > > Hadoop > > > > > > and > > > > > > > >> > JDBC, > > > > > > > >> > > I want to know is there any plan to support for Apache > > > > Cassandra > > > > > > as > > > > > > > >> its > > > > > > > >> > > support CQL and Spark SQL (thru Spark connector) which > are > > > > more > > > > > > SQL > > > > > > > >> like. > > > > > > > >> > > If yes, I wish to know the details and contribute as > well. > > > > > > > >> > > > > > > > > > >> > > Thanks > > > > > > > >> > > Ajay > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
