[jira] [Resolved] (DRILL-2154) Provide an option to change the Parquet compression during CTAS

2015-02-23 Thread Steven Phillips (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Phillips resolved DRILL-2154.

Resolution: Fixed

Fixed in 00c08ef

> Provide an option to change the Parquet compression during CTAS
> ---
>
> Key: DRILL-2154
> URL: https://issues.apache.org/jira/browse/DRILL-2154
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Affects Versions: 0.7.0
>Reporter: Aman Sinha
>Assignee: Steven Phillips
> Fix For: 0.8.0
>
> Attachments: DRILL-2154.patch, DRILL-2154.patch
>
>
> Currently, when doing a CTAS, the Parquet writer has a hardcoded SNAPPY 
> compression.  For testing and debugging purposes we should allow this to be 
> configurable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 30965: Follow up on DRILL-133 (LocalExchange) to save CPU cycles on hash generation when using in HashToLocalExchange

2015-02-23 Thread Yuliya Feldman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30965/
---

(Updated Feb. 23, 2015, 11:32 p.m.)


Review request for drill, Jacques Nadeau, Jinfeng Ni, Steven Phillips, and 
Venki Korukanti.


Changes
---

Addressing review ocmments


Bugs: DRILL-2209
https://issues.apache.org/jira/browse/DRILL-2209


Repository: drill-git


Description
---

Insert Project operator to add new column "EXPRHASH" with hash expression for 
fields that are used for HashToRandomExchange
Remove Project operator after HashRandomExchange (or Demux) since it will 
create problems to fields ordering in HashJoin.

Tight this to MuxExchange - so if MuxExchange is enabled, Project is inserted.


Diffs (updated)
-

  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashToRandomExchangePrel.java
 372c75d 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/30965/diff/


Testing
---

Need to add Unit Tests. tested live, run Functional and TPCH tests


Thanks,

Yuliya Feldman

[jira] [Resolved] (DRILL-2286) Parquet compression causes read errors

2015-02-23 Thread Steven Phillips (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Phillips resolved DRILL-2286.

Resolution: Duplicate

> Parquet compression causes read errors
> --
>
> Key: DRILL-2286
> URL: https://issues.apache.org/jira/browse/DRILL-2286
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 0.8.0
>Reporter: Adam Gilmore
>Assignee: Steven Phillips
>Priority: Critical
>
> From what I can see, since compression has been added to the Parquet writer, 
> reading errors can occur.
> Basically, things like timestamp and decimal are stored as int64 with some 
> metadata.  It appears that when the column is compressed, it tries to read 
> int64s into a vector of timestamp/decimal types, which causes a cast error.
> Here's the JSON file I'm using:
> {code}
> { "a": 1.5 }
> { "a": 3.5 }
> { "a": 1.5 }
> { "a": 2.5 }
> { "a": 1.5 }
> { "a": 5.5 }
> { "a": 1.5 }
> { "a": 6.0 }
> { "a": 1.5 }
> {code}
> Now create a Parquet table like so:
> create table dfs.tmp.test as (select cast(a as decimal(18,8)) from 
> dfs.tmp.`test.json`)
> Now when you try to query it like so:
> {noformat}
> 0: jdbc:drill:zk=local> select * from dfs.tmp.test;
> Query failed: RemoteRpcException: Failure while running fragment., 
> org.apache.drill.exec.vector.NullableDecimal18Vector cannot be cast to 
> org.apache.drill.exec.vector.NullableBigIntVector [ 
> 91e23d42-fa06-4429-b78e-3ff32352e660 on ...:31010 ]
> [ 91e23d42-fa06-4429-b78e-3ff32352e660 on ...:31010 ]
> Error: exception while executing query: Failure while executing query. 
> (state=,code=0)
> {noformat}
> This is the same for timestamps, for example.
> The relevant code is in ColumnReaderFactory whereby if the column chunk is 
> encoded, it creates specific readers based on the type of the column (in this 
> case int64, instead of timestamp/decimal).
> This is pretty severe, as it looks like the compression is enabled by default 
> now.  I do note that with only 1-2 records in the JSON file, it doesn't 
> bother compressing and the queries then work fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DRILL-1062) DRILL does not handle NULLS FIRST/LAST correctly in ORDER BY clause

2015-02-23 Thread Parth Chandra (JIRA) 

 [ 
https://issues.apache.org/jira/browse/DRILL-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Chandra resolved DRILL-1062.
--
Resolution: Fixed

Resolved in commit 5efc7e6

> DRILL does not handle NULLS FIRST/LAST correctly in ORDER BY clause
> ---
>
> Key: DRILL-1062
> URL: https://issues.apache.org/jira/browse/DRILL-1062
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Functions - Drill
>Reporter: Jinfeng Ni
>Assignee: Daniel Barclay (Drill)
> Fix For: 0.8.0
>
> Attachments: DRILL-1062.2.patch, DRILL-1062.3.patch.txt, 
> DRILL-1062.4.patch.txt
>
>
> ORDER BY clause could specify nulls first or nulls last.  Currently, DRILL 
> will always use nulls last policy. 
> select tbl.topping[3].type from 
> dfs.`/Users/jni/work/incubator-drill/exec/ref/target/test-classes/donuts.json`
>  as tbl order by 1 nulls last;
> ++
> |   EXPR$0   |
> ++
> | Chocolate  |
> | Maple  |
> | Powdered Sugar |
> | Powdered Sugar |
> | null   |
> ++
> 5 rows selected (0.156 seconds)
> 0: jdbc:drill:zk=local> select tbl.topping[3].type from 
> dfs.`/Users/jni/work/incubator-drill/exec/ref/target/test-classes/donuts.json`
>  as tbl order by 1 nulls first;
> ++
> |   EXPR$0   |
> ++
> | Chocolate  |
> | Maple  |
> | Powdered Sugar |
> | Powdered Sugar |
> | null   |
> ++
> 5 rows selected (0.186 seconds)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Are we planning a hangout today ?

2015-02-23 Thread Yash Sharma 
Hi Folks,
Are we planning a hangout today ?

Jason - If you would be joining I would like to hear your thoughts on the
Python UDF support.

Regards

Review Request 31313: DRILL-2280: Refactor ValueVector interface & add an abstract ValueVector implementation

2015-02-23 Thread Hanifi Gunes 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31313/
---

Review request for drill, Mehant Baid and Steven Phillips.


Bugs: DRILL-2280
https://issues.apache.org/jira/browse/DRILL-2280


Repository: drill-git


Description
---

DRILL-2280: Refactor ValueVector interface & provide a base ValueVector 
implementation
I will convert DRILL-2280 to an umbrella around ValueVector interface 
refactoring alone. This issue consists of small set of changes fixing misplaced 
methods.

- Added initial documentation.
- Move ValueVector.Accessor#getReader to ValueVector#getReader
- Use of generics for Accessor & Mutator's
- Misc test fixes and minor refacs


Diffs
-

  exec/java-exec/src/main/codegen/templates/EventBasedRecordWriter.java 
e76178a59bd753ae68c9fe9fcf007e3665ca4174 
  exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 
52a38680aa1e7a1d5aa6e4f5d7e38337fb00783c 
  exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 
ba7c629bf4fe4d2b04f52dcb02e98b532128dc73 
  exec/java-exec/src/main/codegen/templates/RecordValueAccessor.java 
47197315e9e45ef2b5e456e957c7914d22a51551 
  exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 
d39040e4731d645efc411aecf79fba69c50c4f98 
  exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java 
f854d32c59e5be6de60b34bbc4895b3c832c98bc 
  
exec/java-exec/src/main/java/org/apache/drill/exec/expr/EvaluationVisitor.java 
a5758fba9a0e3fa53a6ca7c3b0cca1b322d7 
  
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ColumnReader.java
 2424fac7e9451bc0c92c6455ab1571786bde8572 
  
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableColumnReader.java
 c7ada6bffdea20146511150424068b33f7d4fc65 
  
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet2/DrillParquetReader.java
 8778ef8b88a13263213a9a59679fbd8a1fa999ca 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseDataValueVector.java
 bc3edaae7b0c993845a6a7324240e17a646c0c74 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseValueVector.java 
18da67d5f9900d4fffa2a782cb60cef9289aa72c 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java 
f6644bddbd239fd5d710d73a0dc7bcb2acb1b7e8 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/ObjectVector.java 
3c15db314184ff33f48bb7c5515b47dd98579814 
  exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueVector.java 
df6a4860b8cd3f0ae6b102a98ed610304a0081dd 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractContainerVector.java
 3bc7d11b9d7bf7f3159041ac2235333519d17931 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java
 c5dc5ba2537f249adf524844836320f4004d78ba 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java
 131f2a3354a4162b22927bd9e464f32487ed2af2 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java
 ad8c66fe6c11c01ded1c651577015ec16ac8ea70 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedListReaderImpl.java
 c51dfda69a568e3877e1edb75b179870987ecff8 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedMapReaderImpl.java
 91362770c3265104876f772f4f50441b6e33dc4d 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/SingleListReaderImpl.java
 40fa6d46fe17ac1fca013980fc2c4381c4a48abc 
  
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/SingleMapReaderImpl.java
 5c8f688d96661d4276f84c246138fd9c03523591 
  exec/java-exec/src/test/java/org/apache/drill/exec/expr/ExpressionTest.java 
2311cfd830b78a5334848b451688bb5d6489e372 
  
exec/java-exec/src/test/java/org/apache/drill/exec/vector/complex/writer/TestRepeated.java
 feaef4d3f6694955012e63cd6ecc03278a6e5882 

Diff: https://reviews.apache.org/r/31313/diff/


Testing
---

unit tests.


Thanks,

Hanifi Gunes

Re: Review Request 30965: Follow up on DRILL-133 (LocalExchange) to save CPU cycles on hash generation when using in HashToLocalExchange

2015-02-23 Thread Yuliya Feldman 


> On Feb. 23, 2015, 5:44 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java,
> >  line 112
> > 
> >
> > MuxExchange has Project as its child. So, MuxExchange will have same 
> > traits as Project (addColumnprojectPrel), in stead of its parent (prel).

Will definitely fix it - thank you for pointing out


> On Feb. 23, 2015, 5:44 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java,
> >  line 127
> > 
> >
> > I'm not fully clear about the motification of inserting the hash 
> > expression into Project. But here if we remove the compuated hash 
> > expression, does it mean that the down stream operator will not be able to 
> > refer to this computed value, and have to re-compute?

The problem is that if we have HashJoin later on it is not aware of additional 
column and it will be failing, so after discussion with Jacques we decided to 
add Project before HashExchage and remove it after - so to thw world outside of 
Mux/HashExchange/Demux it will look as Project was never inserted


- Yuliya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30965/#review73732
---


On Feb. 23, 2015, 4:09 p.m., Yuliya Feldman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30965/
> ---
> 
> (Updated Feb. 23, 2015, 4:09 p.m.)
> 
> 
> Review request for drill, Jacques Nadeau, Jinfeng Ni, Steven Phillips, and 
> Venki Korukanti.
> 
> 
> Bugs: DRILL-2209
> https://issues.apache.org/jira/browse/DRILL-2209
> 
> 
> Repository: drill-git
> 
> 
> Description
> ---
> 
> Insert Project operator to add new column "EXPRHASH" with hash expression for 
> fields that are used for HashToRandomExchange
> Remove Project operator after HashRandomExchange (or Demux) since it will 
> create problems to fields ordering in HashJoin.
> 
> Tight this to MuxExchange - so if MuxExchange is enabled, Project is inserted.
> 
> 
> Diffs
> -
> 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashToRandomExchangePrel.java
>  372c75d 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30965/diff/
> 
> 
> Testing
> ---
> 
> Need to add Unit Tests. tested live, run Functional and TPCH tests
> 
> 
> Thanks,
> 
> Yuliya Feldman
> 
>

Re: Review Request 30965: Follow up on DRILL-133 (LocalExchange) to save CPU cycles on hash generation when using in HashToLocalExchange

2015-02-23 Thread Jinfeng Ni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30965/#review73732
---



exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java


MuxExchange has Project as its child. So, MuxExchange will have same traits 
as Project (addColumnprojectPrel), in stead of its parent (prel).



exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java


I'm not fully clear about the motification of inserting the hash expression 
into Project. But here if we remove the compuated hash expression, does it mean 
that the down stream operator will not be able to refer to this computed value, 
and have to re-compute?


- Jinfeng Ni


On Feb. 23, 2015, 4:09 p.m., Yuliya Feldman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30965/
> ---
> 
> (Updated Feb. 23, 2015, 4:09 p.m.)
> 
> 
> Review request for drill, Jacques Nadeau, Jinfeng Ni, Steven Phillips, and 
> Venki Korukanti.
> 
> 
> Bugs: DRILL-2209
> https://issues.apache.org/jira/browse/DRILL-2209
> 
> 
> Repository: drill-git
> 
> 
> Description
> ---
> 
> Insert Project operator to add new column "EXPRHASH" with hash expression for 
> fields that are used for HashToRandomExchange
> Remove Project operator after HashRandomExchange (or Demux) since it will 
> create problems to fields ordering in HashJoin.
> 
> Tight this to MuxExchange - so if MuxExchange is enabled, Project is inserted.
> 
> 
> Diffs
> -
> 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashToRandomExchangePrel.java
>  372c75d 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30965/diff/
> 
> 
> Testing
> ---
> 
> Need to add Unit Tests. tested live, run Functional and TPCH tests
> 
> 
> Thanks,
> 
> Yuliya Feldman
> 
>

Re: Using CTAS with nested structures

2015-02-23 Thread rahul challapalli 
I will raise an enhancement request for this. Thank You.

- Rahul

On Mon, Feb 23, 2015 at 3:35 PM, Adam Gilmore  wrote:

> I submitted a patch for reading all JSON numbers as doubles; however, it'd
> probably be nice to extend that to specify a default to read as anything.
>
> Something like ...
>
> alter session set `store.json.read_numbers_as` = 'DECIMAL(5,2)';
>
> would be useful.
>
> On Tue, Feb 24, 2015 at 9:32 AM, Steven Phillips 
> wrote:
>
> > I don't think this is supported currently.
> >
> > On Mon, Feb 23, 2015 at 3:29 PM, rahul challapalli <
> > challapallira...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I want to able to read from a nested json file, cast individual fields,
> > and
> > > still be able to maintain the nested structure. If this is possible,
> how
> > do
> > > I compose my CTAS query?
> > >
> > > Example :
> > >
> > > {
> > >   "map" : {"col1":12.2, "col2":25.5}
> > > }
> > >
> > > The below query generates a flat parquet, but I still want to maintain
> > the
> > > nestedness
> > >
> > > select cast(d.map.col1 as decimal(5,2)) ,  cast(d.map.col2 as double))
> > from
> > > `data.json`;
> > >
> > >
> > > I am looking for something on the below lines :
> > >
> > > select cast(m1 as map(col1:decimal, col2:double)) from `data.json`;
> > >
> > > - Rahul
> > >
> >
> >
> >
> > --
> >  Steven Phillips
> >  Software Engineer
> >
> >  mapr.com
> >
>

Re: Using CTAS with nested structures

2015-02-23 Thread Adam Gilmore 
I submitted a patch for reading all JSON numbers as doubles; however, it'd
probably be nice to extend that to specify a default to read as anything.

Something like ...

alter session set `store.json.read_numbers_as` = 'DECIMAL(5,2)';

would be useful.

On Tue, Feb 24, 2015 at 9:32 AM, Steven Phillips 
wrote:

> I don't think this is supported currently.
>
> On Mon, Feb 23, 2015 at 3:29 PM, rahul challapalli <
> challapallira...@gmail.com> wrote:
>
> > Hi,
> >
> > I want to able to read from a nested json file, cast individual fields,
> and
> > still be able to maintain the nested structure. If this is possible, how
> do
> > I compose my CTAS query?
> >
> > Example :
> >
> > {
> >   "map" : {"col1":12.2, "col2":25.5}
> > }
> >
> > The below query generates a flat parquet, but I still want to maintain
> the
> > nestedness
> >
> > select cast(d.map.col1 as decimal(5,2)) ,  cast(d.map.col2 as double))
> from
> > `data.json`;
> >
> >
> > I am looking for something on the below lines :
> >
> > select cast(m1 as map(col1:decimal, col2:double)) from `data.json`;
> >
> > - Rahul
> >
>
>
>
> --
>  Steven Phillips
>  Software Engineer
>
>  mapr.com
>

Re: Using CTAS with nested structures

2015-02-23 Thread Steven Phillips 
I don't think this is supported currently.

On Mon, Feb 23, 2015 at 3:29 PM, rahul challapalli <
challapallira...@gmail.com> wrote:

> Hi,
>
> I want to able to read from a nested json file, cast individual fields, and
> still be able to maintain the nested structure. If this is possible, how do
> I compose my CTAS query?
>
> Example :
>
> {
>   "map" : {"col1":12.2, "col2":25.5}
> }
>
> The below query generates a flat parquet, but I still want to maintain the
> nestedness
>
> select cast(d.map.col1 as decimal(5,2)) ,  cast(d.map.col2 as double)) from
> `data.json`;
>
>
> I am looking for something on the below lines :
>
> select cast(m1 as map(col1:decimal, col2:double)) from `data.json`;
>
> - Rahul
>



-- 
 Steven Phillips
 Software Engineer

 mapr.com

Using CTAS with nested structures

2015-02-23 Thread rahul challapalli 
Hi,

I want to able to read from a nested json file, cast individual fields, and
still be able to maintain the nested structure. If this is possible, how do
I compose my CTAS query?

Example :

{
  "map" : {"col1":12.2, "col2":25.5}
}

The below query generates a flat parquet, but I still want to maintain the
nestedness

select cast(d.map.col1 as decimal(5,2)) ,  cast(d.map.col2 as double)) from
`data.json`;


I am looking for something on the below lines :

select cast(m1 as map(col1:decimal, col2:double)) from `data.json`;

- Rahul

Re: Review Request 31107: Ability to make PartitionSender multithreaded - useful in case of LocalExchange being enabled, as it allows to deal with high volume of incoming data

2015-02-23 Thread Yuliya Feldman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31107/
---

(Updated Feb. 23, 2015, 3:29 p.m.)


Review request for drill, Chris Westin, Jacques Nadeau, Steven Phillips, and 
Venki Korukanti.


Changes
---

Addressing review comments


Bugs: DRILL-2210
https://issues.apache.org/jira/browse/DRILL-2210


Repository: drill-git


Description
---

In addition to description

Fixed few classes that did not handle multithreading well
Added/Changed some Stats behavior to allow stats merge from multiple threads, 
since again this class is not suitable to be used in multithreaded environment
Introduced new decorator class to handle multi thrteading (or not)  to minimize 
changes to ParitionSenderRootExec class


Diffs (updated)
-

  exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorStats.java 
0e9da0e 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SendingAccountor.java
 7af7b65 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
 f09acaa 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/Partitioner.java
 5ed9c39 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
 PRE-CREATION 
  
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
 4292c09 
  
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
 faa8546 
  
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
 aa0a5ad 

Diff: https://reviews.apache.org/r/31107/diff/


Testing
---

Still need to provide Unit Tests.

Functional tests are passing

Performance tests were run and look promising for some queries


Thanks,

Yuliya Feldman

Review Request 31319: DRILL-2283: Fixed: INFORMATION_SCHEMA strings were VARCHAR(1), breaking comparison

2015-02-23 Thread Daniel Barclay 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31319/
---

Review request for drill, Mehant Baid and Parth Chandra.


Bugs: DRILL-2283
https://issues.apache.org/jira/browse/DRILL-2283


Repository: drill-git


Description
---

Fixed implementation string columns of INFORMATION SCHEMA tables from calling 
method that set type to VARCHAR(1) to setting type to VARCHAR().


Diffs
-

  
exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaTable.java
 c343c5c 
  
exec/java-exec/src/test/java/org/apache/drill/exec/store/ischema/Drill2283InfoSchemaVarchar1BugTest.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/31319/diff/


Testing
---

Ran new specific unit tests and existing test suites.


Thanks,

Daniel Barclay

[jira] [Created] (DRILL-2289) Email is still pointing to the old ( incubator.apache.org) should be u...@drill.apache.org.

2015-02-23 Thread Brahma Reddy Battula (JIRA) 
Brahma Reddy Battula created DRILL-2289:
---

 Summary: Email is still pointing to the old ( 
incubator.apache.org) should be u...@drill.apache.org.
 Key: DRILL-2289
 URL: https://issues.apache.org/jira/browse/DRILL-2289
 Project: Apache Drill
  Issue Type: Bug
  Components: Documentation
Reporter: Brahma Reddy Battula
Assignee: Bridget Bevens


Not sure who to direct this to, but:
http://drill.apache.org/faq/

FAQ:"How can I ask questions and provide feedback?" has the the following

How can I ask questions and provide feedback?
Please post your questions and feedback on  
*{color:red}drill-u...@incubator.apache.org.{color}*  We are happy to have you 
try out Drill and help with any questions!

It should be  *{color:green}u...@drill.apache.org address{color}* 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31107: Ability to make PartitionSender multithreaded - useful in case of LocalExchange being enabled, as it allows to deal with high volume of incoming data

2015-02-23 Thread Chris Westin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31107/#review73593
---



exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java


You could use Thread.currentThread() once here, and then reuse.


- Chris Westin


On Feb. 20, 2015, 5:39 p.m., Yuliya Feldman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31107/
> ---
> 
> (Updated Feb. 20, 2015, 5:39 p.m.)
> 
> 
> Review request for drill, Chris Westin, Jacques Nadeau, Steven Phillips, and 
> Venki Korukanti.
> 
> 
> Bugs: DRILL-2210
> https://issues.apache.org/jira/browse/DRILL-2210
> 
> 
> Repository: drill-git
> 
> 
> Description
> ---
> 
> In addition to description
> 
> Fixed few classes that did not handle multithreading well
> Added/Changed some Stats behavior to allow stats merge from multiple threads, 
> since again this class is not suitable to be used in multithreaded environment
> Introduced new decorator class to handle multi thrteading (or not)  to 
> minimize changes to ParitionSenderRootExec class
> 
> 
> Diffs
> -
> 
>   exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorStats.java 
> 0e9da0e 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SendingAccountor.java
>  7af7b65 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
>  f09acaa 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/Partitioner.java
>  5ed9c39 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
>  PRE-CREATION 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
>  4292c09 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
>  faa8546 
>   
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
>  aa0a5ad 
> 
> Diff: https://reviews.apache.org/r/31107/diff/
> 
> 
> Testing
> ---
> 
> Still need to provide Unit Tests.
> 
> Functional tests are passing
> 
> Performance tests were run and look promising for some queries
> 
> 
> Thanks,
> 
> Yuliya Feldman
> 
>

Re: Review Request 29816: DRILL-92: Cassandra Storage Engine

2015-02-23 Thread Yash Sharma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29816/
---

(Updated Feb. 23, 2015, 9:12 a.m.)


Review request for drill.


Changes
---

Updated pacth to remove unwanted imports causing compilation error.


Repository: drill-git


Description
---

Cassandra storage engine implementation using the datastax Java driver.
Storage plugin format:
{
  "type": "cassandra",
  "host": "localhost",
  "port": 9042,
  "enabled": true
}

Sample Supported Queries:
SELECT id,pog_rank FROM cassandra.recsys.`trending_now` t;
SELECT * FROM cassandra.recsys.`trending_now` t WHERE id = 'id0004' and 
pog_rank = 2;


Known Issues:
- Filter pushdown is broken. Currently only Drill side filters work.


Diffs (updated)
-

  contrib/pom.xml d75dc79 
  contrib/storage-cassandra/pom.xml PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraCompareFunctionsProcessor.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraDatabaseSchema.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraFilterBuilder.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraGroupScan.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraPartitionToken.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraPushDownFilterForScan.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraRecordReader.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraScanBatchCreator.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraScanSpec.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraSchemaFactory.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraStoragePlugin.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraStoragePluginConfig.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraSubScan.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/CassandraUtil.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/DrillCassandraConstants.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/main/java/org/apache/drill/exec/store/cassandra/connection/CassandraConnectionManager.java
 PRE-CREATION 
  contrib/storage-cassandra/src/main/resources/bootstrap-storage-plugins.json 
PRE-CREATION 
  contrib/storage-cassandra/src/main/resources/drill-module.conf PRE-CREATION 
  
contrib/storage-cassandra/src/test/java/org/apache/drill/cassandra/BaseCassandraTest.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/java/org/apache/drill/cassandra/CassandraFilterPushdownTest.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/java/org/apache/drill/cassandra/CassandraProjectPushdownTest.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/java/org/apache/drill/cassandra/CassandraRecordReaderTest.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/java/org/apache/drill/cassandra/CassandraTestConstants.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/java/org/apache/drill/cassandra/CassandraTestSuite.java
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/resources/cassandra-plans/cassandra_scan_screen_physical.json
 PRE-CREATION 
  
contrib/storage-cassandra/src/test/resources/cassandra-plans/cassandra_scan_screen_with_columns_physical.json
 PRE-CREATION 
  distribution/pom.xml 703a4c4 
  distribution/src/assemble/bin.xml 8f6e90d 

Diff: https://reviews.apache.org/r/29816/diff/


Testing
---

Testing done via Sqlline and Web Interface. 
Tested with Physical plan and Sql query via New Test framework.

Testcases:

CassandraRecordReaderTest
- testPlanLocal
- testPlanLocalWithColumns

CassandraProjectPushdownTest
- testProjection

CassandraFilterPushdownTest
- testSelectAll
- testFilter
- testFilter1
- testFilter2
- testFilterZeroRecords
- testFilterWithOrCondition
- testFilterWithAndCondition


Thanks,

Yash Sharma