[jira] [Commented] (CARBONDATA-583) Replace Function is not working for string/char

[sandeep purohit (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15800422#comment-15800422
 ] 

sandeep purohit commented on CARBONDATA-583:


[~nareshpr] yes regexp_replace function already available in 
org.apache.spark.sql.functions package so if we want to use that in place of 
replace then we document it somewhere, Most of time user try replace directly 
as other database.


> Replace Function is not working  for string/char
> 
>
> Key: CARBONDATA-583
> URL: https://issues.apache.org/jira/browse/CARBONDATA-583
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.0.0-incubating
> Environment: cluster
>Reporter: Anurag Srivastava
>Assignee: Rahul Kumar
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I am running "replace" function but it is giving error : "undefined function 
> replace".
> Query : select replace('aaabbccaabb', 'aaa', 't');
> Expected Result : "tbbccaabb"
> Result : Error: org.apache.spark.sql.AnalysisException: undefined function 
> replace; line 1 pos 30 (state=,code=0) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep purohit closed CARBONDATA-591.
--
Resolution: Invalid

> Remove unused code for spark 2.0 datatype utils
> ---
>
> Key: CARBONDATA-591
> URL: https://issues.apache.org/jira/browse/CARBONDATA-591
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: sandeep purohit
>Priority: Trivial
> Attachments: screenshot-1.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Remove unused code for data type utils for spark 2.0. I look the below code 
> snippet and debug that there is code for spark 2.x datatype conversion in  
> DataTypeConverterUtil.scala.
> {code:title=DataTypeConverterUtil.scala|theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
>   def convertToCarbonTypeForSpark2(dataType: String): DataType = {
> dataType.toLowerCase match {
>   case "stringtype" => DataType.STRING
>   case "inttype" => DataType.INT
>   case "integertype" => DataType.INT
>   case "tinyinttype" => DataType.SHORT
>   case "shorttype" => DataType.SHORT
>   case "longtype" => DataType.LONG
>   case "biginttype" => DataType.LONG
>   case "numerictype" => DataType.DOUBLE
>   case "doubletype" => DataType.DOUBLE
>   case "decimaltype" => DataType.DECIMAL
>   case "timestamptype" => DataType.TIMESTAMP
>   case "datetype" => DataType.DATE
>   case "arraytype" => DataType.ARRAY
>   case "structtype" => DataType.STRUCT
>   case _ => sys.error(s"Unsupported data type: $dataType")
> }
> }
> {code}
> In spark 2.x there is types stringtype and inttype etc as a API not in the 
> query itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata issue #491: [CARBONDATA-583] Add replace function suppo...

[phalodi]

Github user phalodi commented on the issue:

https://github.com/apache/incubator-carbondata/pull/491
  
@nareshpr yes regexp_replace function already available in 
org.apache.spark.sql.functions package so if we want to use that in place of 
replace then we document it somewhere, Most of time user try replace directly 
as other database.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #497: [CARBONDATA-591] Remove unused code ...

[phalodi]

Github user phalodi closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/497


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-596) unable to crete filter resolver tree exception when using equals expression with carbon input format api

[anubhav tarar (JIRA)]

anubhav tarar created CARBONDATA-596:


 Summary: unable to crete filter resolver tree exception when using 
equals expression with carbon input format api
 Key: CARBONDATA-596
 URL: https://issues.apache.org/jira/browse/CARBONDATA-596
 Project: CarbonData
  Issue Type: Bug
  Components: hadoop-integration
Affects Versions: 1.0.0-incubating
Reporter: anubhav tarar
Priority: Minor


unable to crete filter resolver tree exception when using equals expression

 Expression expression = new EqualToExpression(new ColumnExpression("c1", 
DataType.STRING),
new LiteralExpression("a", DataType.STRING));


CarbonInputFormat.setFilterPredicates(job.getConfiguration(), 
expression);
List splits = carbonInputFormat.getSplits(job);

it throws java.io.IOException: Error while resolving filter expression




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata issue #450: [CARBONDATA-545]Added support for offheap s...

[QiangCai]

Github user QiangCai commented on the issue:

https://github.com/apache/incubator-carbondata/pull/450
  
@kumarvishal09 
please rebase and fix some known issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #499: [CARBONDATA-218]fix data loading issue for ...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/incubator-carbondata/pull/499
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #499: [CARBONDATA-218]fix data loading iss...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/499


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #499: [CARBONDATA-218]fix data loading issue for ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/499
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/449/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #499: [CARBONDATA-218]fix data loading iss...

[QiangCai]

GitHub user QiangCai opened a pull request:

https://github.com/apache/incubator-carbondata/pull/499

[CARBONDATA-218]fix data loading issue for UT



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/incubator-carbondata 
fixDataLoadingIssue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #499


commit 0cfefbb450b596da23f87a9cab65016c94f96a0a
Author: QiangCai 
Date:   2017-01-05T03:03:25Z

fixDataLoadingIssue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #481: [WIP]reuse test case for integration module

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/481
  
Build Failed  with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/448/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #493: [CARBONDATA-588] cleanup WriterCompressionM...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/493
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/447/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #493: [CARBONDATA-588] cleanup WriterCompressionM...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/493
  
Build Failed  with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/446/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #493: [CARBONDATA-588] cleanup WriterCompressionM...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/493
  
Build Failed  with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/444/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #491: [CARBONDATA-583] Add replace function suppo...

[nareshpr]

Github user nareshpr commented on the issue:

https://github.com/apache/incubator-carbondata/pull/491
  
Can you check regexp_replace spark function? This function replace all 
substrings of the specified string that matches the pattern


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #498: [CARBONDATA-568][Minor][Follow-Up] c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/498


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #498: [CARBONDATA-568][Minor][Follow-Up] clean up...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/incubator-carbondata/pull/498
  
LGTM



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #498: [CARBONDATA-568][Minor][Follow-Up] clean up...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/incubator-carbondata/pull/498
  
@jackylk please take a look, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #498: [CARBONDATA-568][Minor][Follow-Up] c...

[zzcclp]

GitHub user zzcclp opened a pull request:

https://github.com/apache/incubator-carbondata/pull/498

[CARBONDATA-568][Minor][Follow-Up] clean up code for carbon-core module

using "new java.util.LinkedHashSet" instead of "new util.LinkedHashSet"



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zzcclp/incubator-carbondata cleancore-followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #498


commit 109d3833b5e9fd4ed9c2f231145c149be71903a5
Author: Zhang Zhichao <441586...@qq.com>
Date:   2017-01-04T16:03:41Z

[CARBONDATA-568][Follow-Up] clean up code for carbon-core module

using "new java.util.LinkedHashSet" instead of "new util.LinkedHashSet"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (CARBONDATA-595) Drop Table for carbon throws NPE with HDFS lock type.

[Babulal (JIRA)]

Babulal created CARBONDATA-595:
--

 Summary: Drop Table for carbon throws NPE with HDFS lock type.
 Key: CARBONDATA-595
 URL: https://issues.apache.org/jira/browse/CARBONDATA-595
 Project: CarbonData
  Issue Type: Bug
Affects Versions: 0.2.0-incubating
Reporter: Babulal
Priority: Minor


Start version :- 1.6.2 
Start carbon thrift server
set HDFS LOCK Type

drop table from beeline
0: jdbc:hive2://hacluster> drop table oscon_new_1;
Error: java.lang.NullPointerException (state=,code=0)

Error in thrftserver 

17/01/04 20:40:08 AUDIT DropTableCommand: 
[hadoop-master][anonymous][Thread-182]Deleted table [oscon_new_1] under 
database [default]
17/01/04 20:40:08 ERROR AbstractDFSCarbonFile: pool-25-thread-12 Exception 
occured:File does not exist: 
hdfs://hacluster/opt/CarbonStore/default/oscon_new_1/droptable.lock
17/01/04 20:40:08 ERROR SparkExecuteStatementOperation: Error executing query, 
currentState RUNNING,
java.lang.NullPointerException
at 
org.apache.carbondata.core.datastorage.store.filesystem.AbstractDFSCarbonFile.delete(AbstractDFSCarbonFile.java:128)
at 
org.apache.carbondata.lcm.locks.HdfsFileLock.unlock(HdfsFileLock.java:110)
at 
org.apache.spark.sql.execution.command.DropTableCommand.run(carbonTableSchema.scala:613)
at 
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at 
org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)



Note :- lock file and data are deleted successfully but in beeline it throws 
ERROR message instead of success. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata pull request #492: [CARBONDATA-440] Providing the updat...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/492#discussion_r94584912
  
--- Diff: 
core/src/main/java/org/apache/carbondata/common/iudprocessor/cache/DeleteDeltaDataCache.java
 ---
@@ -0,0 +1,29 @@
+package org.apache.carbondata.common.iudprocessor.cache;
+
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Created by S71955 on 06-10-2016.
+ */
+public class DeleteDeltaDataCache {
--- End diff --

Here too can we rename the class to DeleteDeltaCache?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #492: [CARBONDATA-440] Providing the updat...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/492#discussion_r94584731
  
--- Diff: 
core/src/main/java/org/apache/carbondata/common/iudprocessor/cache/BlockletLevelDeleteDeltaDataCache.java
 ---
@@ -0,0 +1,29 @@
+package org.apache.carbondata.common.iudprocessor.cache;
+
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Created by S71955 on 06-10-2016.
+ */
+public class BlockletLevelDeleteDeltaDataCache {
--- End diff --

As an opinion I think it would be better if we can rename the class to 
BlockletLevelDeleteDeltaCache


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #492: [CARBONDATA-440] Providing the update and d...

[manishgupta88]

Github user manishgupta88 commented on the issue:

https://github.com/apache/incubator-carbondata/pull/492
  
Please handle the below things in general.
1. Add apache license to newly added files.
2. Remove the author name as class level comment.
3. Add a basic comment for newly added class that defines the 
responsibility of class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #472: [CARBONDATA-568] clean up code for c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/472


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #482: [CARBONDATA-552] Catch the FilterUns...

[manishgupta88]

Github user manishgupta88 commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/482#discussion_r94582561
  
--- Diff: 
core/src/main/java/org/apache/carbondata/scan/filter/resolver/RowLevelRangeFilterResolverImpl.java
 ---
@@ -187,7 +185,11 @@ public void resolve(AbsoluteTableIdentifier 
absoluteTableIdentifier) {
   
dimColumnEvaluatorInfo.setDimension(columnExpression.getDimension());
   dimColumnEvaluatorInfo.setDimensionExistsInCurrentSilce(false);
   if 
(columnExpression.getDimension().hasEncoding(Encoding.DIRECT_DICTIONARY)) {
-
filterInfo.setFilterList(getDirectSurrogateValues(columnExpression));
+try {
+  
filterInfo.setFilterList(getDirectSurrogateValues(columnExpression));
+} catch (FilterUnsupportedException e) {
+  FilterUtil.logFilterError(e, false);
--- End diff --

In the earlier code, if any exception was thrown it was not caught and 
delegated back to the caller but that behavior will be changed with this 
modified code. So in the catch block please rethrow the exception so that the 
behavior remains the same as old one


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (CARBONDATA-593) Select command seems to be not working on carbon-spark-shell . It throws a runtime error on select query after show method is invoked

[DEEPAK MEHRA (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DEEPAK MEHRA updated CARBONDATA-593:

Attachment: employee.csv

> Select command seems to be not working on carbon-spark-shell . It throws a 
> runtime error on select query after show method is invoked
> -
>
> Key: CARBONDATA-593
> URL: https://issues.apache.org/jira/browse/CARBONDATA-593
> Project: CarbonData
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 1.0.0-incubating
>Reporter: DEEPAK MEHRA
>Priority: Minor
> Attachments: employee.csv
>
>
> Select command seems to be not working on carbon-spark-shell . It throws a 
> runtime error on select query after show method is invoked. It says 
> java.io.FileNotFoundException: File does not exist: 
> /home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0
> Query Executed :
> scala> cc.sql("create table demo(id int,name string,age int,city string) 
> stored by 'carbondata'").show
> scala> cc.sql("LOAD DATA inpath 
> 'hdfs://hadoop-master:54311/data/employee.csv' INTO table demo");
>  cc.sql("select * from demo").show
> Result :It's throwing java.lang.RuntimeException : File Not Found
> scala> cc.sql("select * from mydb.demo").show
> INFO  04-01 16:29:22,157 - main Query [SELECT * FROM MYDB.DEMO]
> INFO  04-01 16:29:22,160 - Parsing command: select * from mydb.demo
> INFO  04-01 16:29:22,161 - Parse Completed
> INFO  04-01 16:29:22,162 - Parsing command: select * from mydb.demo
> INFO  04-01 16:29:22,163 - Parse Completed
> INFO  04-01 16:29:22,163 - 0: get_table : db=mydb tbl=demo
> INFO  04-01 16:29:22,163 - ugi=hduser ip=unknown-ip-addr  cmd=get_table : 
> db=mydb tbl=demo
> INFO  04-01 16:29:22,195 - main Starting to optimize plan
> java.io.FileNotFoundException: File does not exist: 
> /home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
>   at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$15.(DistributedFileSystem.java:726)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:717)
>   at 
> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1780)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getFileStatusOfSegments(CarbonInputFormat.java:559)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.listStatus(CarbonInputFormat.java:519)
>   at 
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getSplitsInternal(CarbonInputFormat.java:251)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getTableBlockInfo(CarbonInputFormat.java:372)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getSegmentAbstractIndexs(CarbonInputFormat.java:402)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getDataBlocksOfSegment(CarbonInputFormat.java:325)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getSplits(CarbonInputFormat.java:288)
>   at 
> org.apache.carbondata.hadoop.CarbonInputFormat.getSplits(CarbonInputFormat.java:237)
>   at 
> org.apache.carbondata.spark.rdd.CarbonScanRDD.getPartitions(CarbonScanRDD.scala:82)
>   at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
>   at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
>   at scala.Option.getOrElse(Option.scala:120)
>   at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
>   at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
>   at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
>   at scala.Option.getOrElse(Option.scala:120)
>   at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
>   at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
>   at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
>   at scala.Option.getOrElse(Option.scala:120)
>   at 

[jira] [Updated] (CARBONDATA-593) Select command seems to be not working on carbon-spark-shell . It throws a runtime error on select query after show method is invoked

[DEEPAK MEHRA (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DEEPAK MEHRA updated CARBONDATA-593:

Description: 
Select command seems to be not working on carbon-spark-shell . It throws a 
runtime error on select query after show method is invoked. It says 
java.io.FileNotFoundException: File does not exist: 
/home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0


Query Executed :
scala> cc.sql("create table demo(id int,name string,age int,city string) stored 
by 'carbondata'").show

scala> cc.sql("LOAD DATA inpath 'hdfs://hadoop-master:54311/data/employee.csv' 
INTO table demo");


 cc.sql("select * from demo").show
Result :It's throwing java.lang.RuntimeException : File Not Found


scala> cc.sql("select * from mydb.demo").show
INFO  04-01 16:29:22,157 - main Query [SELECT * FROM MYDB.DEMO]
INFO  04-01 16:29:22,160 - Parsing command: select * from mydb.demo
INFO  04-01 16:29:22,161 - Parse Completed
INFO  04-01 16:29:22,162 - Parsing command: select * from mydb.demo
INFO  04-01 16:29:22,163 - Parse Completed
INFO  04-01 16:29:22,163 - 0: get_table : db=mydb tbl=demo
INFO  04-01 16:29:22,163 - ugi=hduser   ip=unknown-ip-addr  cmd=get_table : 
db=mydb tbl=demo
INFO  04-01 16:29:22,195 - main Starting to optimize plan
java.io.FileNotFoundException: File does not exist: 
/home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0
at 
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:747)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$15.(DistributedFileSystem.java:726)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:717)
at 
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1780)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getFileStatusOfSegments(CarbonInputFormat.java:559)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.listStatus(CarbonInputFormat.java:519)
at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getSplitsInternal(CarbonInputFormat.java:251)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getTableBlockInfo(CarbonInputFormat.java:372)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getSegmentAbstractIndexs(CarbonInputFormat.java:402)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getDataBlocksOfSegment(CarbonInputFormat.java:325)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getSplits(CarbonInputFormat.java:288)
at 
org.apache.carbondata.hadoop.CarbonInputFormat.getSplits(CarbonInputFormat.java:237)
at 
org.apache.carbondata.spark.rdd.CarbonScanRDD.getPartitions(CarbonScanRDD.scala:82)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at 
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at 
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at 
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at 
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
  

[GitHub] incubator-carbondata issue #480: [Carbondata 390] float data type supported

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/incubator-carbondata/pull/480
  
rename title from [Carbondata 390] to [CARBONDATA-390]


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #480: [Carbondata 390] float data type sup...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/480#discussion_r94580570
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/datastorage/store/compression/nondecimal/UnCompressNonDecimalByte.java
 ---
@@ -85,12 +85,15 @@
   }
 
   @Override public CarbonReadDataHolder getValues(int decimal, Object 
maxValueObject) {
--- End diff --

This is not the way you supposed to do, please check how long data type is 
supported.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (CARBONDATA-592) In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is showing error for that in logs

[Payal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Payal updated CARBONDATA-592:
-
Description: 
In Single Pass loading , When we Execute the load query without 
ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
 main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan not 
be used together, and USE_KETTLE must be set as false

For Example: /CREATE TABLE
CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 
bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');

//LOAD DATA

LOAD DATA INPATH 'hdfs://localhost:54310/Huawei/uniqdata/7000_UniqData.csv' 
into table uniq_include_dictionary OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true'


Logs as attached in Screenshot.


  was:
In Single Pas loading , When we Execute the load query without 
ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
 main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan not 
be used together, and USE_KETTLE must be set as false

For Example: /CREATE TABLE
CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 
bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');

//LOAD DATA

LOAD DATA INPATH 'hdfs://localhost:54310/Huawei/uniqdata/7000_UniqData.csv' 
into table uniq_include_dictionary OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true'






> In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is 
> showing error for that in logs
> --
>
> Key: CARBONDATA-592
> URL: https://issues.apache.org/jira/browse/CARBONDATA-592
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.0.0-incubating
>Reporter: Payal
>Priority: Minor
> Attachments: 7000_UniqData.csv, single_pass.png
>
>
> In Single Pass loading , When we Execute the load query without 
> ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
>  main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan 
> not be used together, and USE_KETTLE must be set as false
> For Example: /CREATE TABLE
> CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> //LOAD DATA
> LOAD DATA INPATH 'hdfs://localhost:54310/Huawei/uniqdata/7000_UniqData.csv' 
> into table uniq_include_dictionary OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true'
> Logs as attached in Screenshot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-592) In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is showing error for that in logs

[Payal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Payal updated CARBONDATA-592:
-
Attachment: 7000_UniqData.csv

> In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is 
> showing error for that in logs
> --
>
> Key: CARBONDATA-592
> URL: https://issues.apache.org/jira/browse/CARBONDATA-592
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.0.0-incubating
>Reporter: Payal
>Priority: Minor
> Attachments: 7000_UniqData.csv, single_pass.png
>
>
> In Single Pas loading , When we Execute the load query without 
> ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
>  main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan 
> not be used together, and USE_KETTLE must be set as false
> For Example: /CREATE TABLE
> CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> //LOAD DATA
> LOAD DATA INPATH 'hdfs://localhost:54310/Huawei/uniqdata/7000_UniqData.csv' 
> into table uniq_include_dictionary OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-592) In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is showing error for that in logs

[Payal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Payal updated CARBONDATA-592:
-
Description: 
In Single Pas loading , When we Execute the load query without 
ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
 main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan not 
be used together, and USE_KETTLE must be set as false

For Example: /CREATE TABLE
CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 
bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');

//LOAD DATA

LOAD DATA INPATH 'hdfs://localhost:54310/Huawei/uniqdata/7000_UniqData.csv' 
into table uniq_include_dictionary OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true'





  was:
In Single Pas loading , When we Execute the load query without 
ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
 main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan not 
be used together, and USE_KETTLE must be set as false


> In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is 
> showing error for that in logs
> --
>
> Key: CARBONDATA-592
> URL: https://issues.apache.org/jira/browse/CARBONDATA-592
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.0.0-incubating
>Reporter: Payal
>Priority: Minor
> Attachments: single_pass.png
>
>
> In Single Pas loading , When we Execute the load query without 
> ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
>  main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan 
> not be used together, and USE_KETTLE must be set as false
> For Example: /CREATE TABLE
> CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> //LOAD DATA
> LOAD DATA INPATH 'hdfs://localhost:54310/Huawei/uniqdata/7000_UniqData.csv' 
> into table uniq_include_dictionary OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata pull request #496: [Carbondata-591] Remove unused datat...

[phalodi]

Github user phalodi closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/496


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/442/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (CARBONDATA-594) Select command seems to be not working on carbon-spark-shell . It throws a runtime error on select query after show method is invoked

[DEEPAK MEHRA (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

DEEPAK MEHRA closed CARBONDATA-594.
---
Resolution: Duplicate

Duplicate issue .Accidentally raised twice.

> Select command seems to be not working on carbon-spark-shell . It throws a 
> runtime error on select query after show method is invoked
> -
>
> Key: CARBONDATA-594
> URL: https://issues.apache.org/jira/browse/CARBONDATA-594
> Project: CarbonData
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 1.0.0-incubating
>Reporter: DEEPAK MEHRA
>Priority: Minor
>
> Select command seems to be not working on carbon-spark-shell . It throws a 
> runtime error on select query after show method is invoked. It says 
> java.io.FileNotFoundException: File does not exist: 
> /home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-592) In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is showing error for that in logs

[Payal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Payal updated CARBONDATA-592:
-
Affects Version/s: 1.0.0-incubating
  Component/s: data-load

> In Single pass loading ,when we not set ALL_DICTIONARY_PATH still it is 
> showing error for that in logs
> --
>
> Key: CARBONDATA-592
> URL: https://issues.apache.org/jira/browse/CARBONDATA-592
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.0.0-incubating
>Reporter: Payal
>Priority: Minor
> Attachments: single_pass.png
>
>
> In Single Pas loading , When we Execute the load query without 
> ALL_DICTIONARY_PATH with single_pass true it is showing the ERROR ---
>  main Can't use single_pass, because SINGLE_PASS and ALL_DICTIONARY_PATHcan 
> not be used together, and USE_KETTLE must be set as false



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep purohit updated CARBONDATA-591:
---
Description: 
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.

{code:title=DataTypeConverterUtil.scala|theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
  def convertToCarbonTypeForSpark2(dataType: String): DataType = {
dataType.toLowerCase match {
  case "stringtype" => DataType.STRING
  case "inttype" => DataType.INT
  case "integertype" => DataType.INT
  case "tinyinttype" => DataType.SHORT
  case "shorttype" => DataType.SHORT
  case "longtype" => DataType.LONG
  case "biginttype" => DataType.LONG
  case "numerictype" => DataType.DOUBLE
  case "doubletype" => DataType.DOUBLE
  case "decimaltype" => DataType.DECIMAL
  case "timestamptype" => DataType.TIMESTAMP
  case "datetype" => DataType.DATE
  case "arraytype" => DataType.ARRAY
  case "structtype" => DataType.STRUCT
  case _ => sys.error(s"Unsupported data type: $dataType")
}
}
{code}

In spark 2.x there is types stringtype and inttype etc as a API not in the 
query itself.

  was:
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.

{code:title=This is my 
title|theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
  def convertToCarbonTypeForSpark2(dataType: String): DataType = {
dataType.toLowerCase match {
  case "stringtype" => DataType.STRING
  case "inttype" => DataType.INT
  case "integertype" => DataType.INT
  case "tinyinttype" => DataType.SHORT
  case "shorttype" => DataType.SHORT
  case "longtype" => DataType.LONG
  case "biginttype" => DataType.LONG
  case "numerictype" => DataType.DOUBLE
  case "doubletype" => DataType.DOUBLE
  case "decimaltype" => DataType.DECIMAL
  case "timestamptype" => DataType.TIMESTAMP
  case "datetype" => DataType.DATE
  case "arraytype" => DataType.ARRAY
  case "structtype" => DataType.STRUCT
  case _ => sys.error(s"Unsupported data type: $dataType")
}
}
{code}

In spark 2.x there is types stringtype and inttype etc as a API not in the 
query itself.


> Remove unused code for spark 2.0 datatype utils
> ---
>
> Key: CARBONDATA-591
> URL: https://issues.apache.org/jira/browse/CARBONDATA-591
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: sandeep purohit
>Priority: Trivial
> Attachments: screenshot-1.png
>
>
> Remove unused code for data type utils for spark 2.0. I look the below code 
> snippet and debug that there is code for spark 2.x datatype conversion in  
> DataTypeConverterUtil.scala.
> {code:title=DataTypeConverterUtil.scala|theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
>   def convertToCarbonTypeForSpark2(dataType: String): DataType = {
> dataType.toLowerCase match {
>   case "stringtype" => DataType.STRING
>   case "inttype" => DataType.INT
>   case "integertype" => DataType.INT
>   case "tinyinttype" => DataType.SHORT
>   case "shorttype" => DataType.SHORT
>   case "longtype" => DataType.LONG
>   case "biginttype" => DataType.LONG
>   case "numerictype" => DataType.DOUBLE
>   case "doubletype" => DataType.DOUBLE
>   case "decimaltype" => DataType.DECIMAL
>   case "timestamptype" => DataType.TIMESTAMP
>   case "datetype" => DataType.DATE
>   case "arraytype" => DataType.ARRAY
>   case "structtype" => DataType.STRUCT
>   case _ => sys.error(s"Unsupported data type: $dataType")
> }
> }
> {code}
> In spark 2.x there is types stringtype and inttype etc as a API not in the 
> query itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep purohit updated CARBONDATA-591:
---
Description: 
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.

{code:title=This is my 
title|theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
  def convertToCarbonTypeForSpark2(dataType: String): DataType = {
dataType.toLowerCase match {
  case "stringtype" => DataType.STRING
  case "inttype" => DataType.INT
  case "integertype" => DataType.INT
  case "tinyinttype" => DataType.SHORT
  case "shorttype" => DataType.SHORT
  case "longtype" => DataType.LONG
  case "biginttype" => DataType.LONG
  case "numerictype" => DataType.DOUBLE
  case "doubletype" => DataType.DOUBLE
  case "decimaltype" => DataType.DECIMAL
  case "timestamptype" => DataType.TIMESTAMP
  case "datetype" => DataType.DATE
  case "arraytype" => DataType.ARRAY
  case "structtype" => DataType.STRUCT
  case _ => sys.error(s"Unsupported data type: $dataType")
}
}
{code}

In spark 2.x there is types stringtype and inttype etc as a API not in the 
query itself.

  was:
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.

  def convertToCarbonTypeForSpark2(dataType: String): DataType = {
dataType.toLowerCase match {
  case "stringtype" => DataType.STRING
  case "inttype" => DataType.INT
  case "integertype" => DataType.INT
  case "tinyinttype" => DataType.SHORT
  case "shorttype" => DataType.SHORT
  case "longtype" => DataType.LONG
  case "biginttype" => DataType.LONG
  case "numerictype" => DataType.DOUBLE
  case "doubletype" => DataType.DOUBLE
  case "decimaltype" => DataType.DECIMAL
  case "timestamptype" => DataType.TIMESTAMP
  case "datetype" => DataType.DATE
  case "arraytype" => DataType.ARRAY
  case "structtype" => DataType.STRUCT
  case _ => sys.error(s"Unsupported data type: $dataType")
}
}

In spark 2.x there is types stringtype and inttype etc as a API not in the 
query itself.


> Remove unused code for spark 2.0 datatype utils
> ---
>
> Key: CARBONDATA-591
> URL: https://issues.apache.org/jira/browse/CARBONDATA-591
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: sandeep purohit
>Priority: Trivial
> Attachments: screenshot-1.png
>
>
> Remove unused code for data type utils for spark 2.0. I look the below code 
> snippet and debug that there is code for spark 2.x datatype conversion in  
> DataTypeConverterUtil.scala.
> {code:title=This is my 
> title|theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
>   def convertToCarbonTypeForSpark2(dataType: String): DataType = {
> dataType.toLowerCase match {
>   case "stringtype" => DataType.STRING
>   case "inttype" => DataType.INT
>   case "integertype" => DataType.INT
>   case "tinyinttype" => DataType.SHORT
>   case "shorttype" => DataType.SHORT
>   case "longtype" => DataType.LONG
>   case "biginttype" => DataType.LONG
>   case "numerictype" => DataType.DOUBLE
>   case "doubletype" => DataType.DOUBLE
>   case "decimaltype" => DataType.DECIMAL
>   case "timestamptype" => DataType.TIMESTAMP
>   case "datetype" => DataType.DATE
>   case "arraytype" => DataType.ARRAY
>   case "structtype" => DataType.STRUCT
>   case _ => sys.error(s"Unsupported data type: $dataType")
> }
> }
> {code}
> In spark 2.x there is types stringtype and inttype etc as a API not in the 
> query itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep purohit updated CARBONDATA-591:
---
Description: 
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.

  def convertToCarbonTypeForSpark2(dataType: String): DataType = {
dataType.toLowerCase match {
  case "stringtype" => DataType.STRING
  case "inttype" => DataType.INT
  case "integertype" => DataType.INT
  case "tinyinttype" => DataType.SHORT
  case "shorttype" => DataType.SHORT
  case "longtype" => DataType.LONG
  case "biginttype" => DataType.LONG
  case "numerictype" => DataType.DOUBLE
  case "doubletype" => DataType.DOUBLE
  case "decimaltype" => DataType.DECIMAL
  case "timestamptype" => DataType.TIMESTAMP
  case "datetype" => DataType.DATE
  case "arraytype" => DataType.ARRAY
  case "structtype" => DataType.STRUCT
  case _ => sys.error(s"Unsupported data type: $dataType")
}
}

In spark 2.x there is types stringtype and inttype etc as a API not in the 
query itself.

  was:
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.






> Remove unused code for spark 2.0 datatype utils
> ---
>
> Key: CARBONDATA-591
> URL: https://issues.apache.org/jira/browse/CARBONDATA-591
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: sandeep purohit
>Priority: Trivial
> Attachments: screenshot-1.png
>
>
> Remove unused code for data type utils for spark 2.0. I look the below code 
> snippet and debug that there is code for spark 2.x datatype conversion in  
> DataTypeConverterUtil.scala.
>   def convertToCarbonTypeForSpark2(dataType: String): DataType = {
> dataType.toLowerCase match {
>   case "stringtype" => DataType.STRING
>   case "inttype" => DataType.INT
>   case "integertype" => DataType.INT
>   case "tinyinttype" => DataType.SHORT
>   case "shorttype" => DataType.SHORT
>   case "longtype" => DataType.LONG
>   case "biginttype" => DataType.LONG
>   case "numerictype" => DataType.DOUBLE
>   case "doubletype" => DataType.DOUBLE
>   case "decimaltype" => DataType.DECIMAL
>   case "timestamptype" => DataType.TIMESTAMP
>   case "datetype" => DataType.DATE
>   case "arraytype" => DataType.ARRAY
>   case "structtype" => DataType.STRUCT
>   case _ => sys.error(s"Unsupported data type: $dataType")
> }
> }
> In spark 2.x there is types stringtype and inttype etc as a API not in the 
> query itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep purohit updated CARBONDATA-591:
---
Description: 
Remove unused code for data type utils for spark 2.0. I look the below code 
snippet and debug that there is code for spark 2.x datatype conversion in  
DataTypeConverterUtil.scala.





  was:Remove unused code for data type utils for spark 2.0


> Remove unused code for spark 2.0 datatype utils
> ---
>
> Key: CARBONDATA-591
> URL: https://issues.apache.org/jira/browse/CARBONDATA-591
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: sandeep purohit
>Priority: Trivial
> Attachments: screenshot-1.png
>
>
> Remove unused code for data type utils for spark 2.0. I look the below code 
> snippet and debug that there is code for spark 2.x datatype conversion in  
> DataTypeConverterUtil.scala.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-594) Select command seems to be not working on carbon-spark-shell . It throws a runtime error on select query after show method is invoked

[DEEPAK MEHRA (JIRA)]

DEEPAK MEHRA created CARBONDATA-594:
---

 Summary: Select command seems to be not working on 
carbon-spark-shell . It throws a runtime error on select query after show 
method is invoked
 Key: CARBONDATA-594
 URL: https://issues.apache.org/jira/browse/CARBONDATA-594
 Project: CarbonData
  Issue Type: Bug
  Components: sql
Affects Versions: 1.0.0-incubating
Reporter: DEEPAK MEHRA
Priority: Minor


Select command seems to be not working on carbon-spark-shell . It throws a 
runtime error on select query after show method is invoked. It says 
java.io.FileNotFoundException: File does not exist: 
/home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-593) Select command seems to be not working on carbon-spark-shell . It throws a runtime error on select query after show method is invoked

[DEEPAK MEHRA (JIRA)]

DEEPAK MEHRA created CARBONDATA-593:
---

 Summary: Select command seems to be not working on 
carbon-spark-shell . It throws a runtime error on select query after show 
method is invoked
 Key: CARBONDATA-593
 URL: https://issues.apache.org/jira/browse/CARBONDATA-593
 Project: CarbonData
  Issue Type: Bug
  Components: sql
Affects Versions: 1.0.0-incubating
Reporter: DEEPAK MEHRA
Priority: Minor


Select command seems to be not working on carbon-spark-shell . It throws a 
runtime error on select query after show method is invoked. It says 
java.io.FileNotFoundException: File does not exist: 
/home/hduser/software/spark-1.6.2-bin-hadoop2.6/carbon.store/mydb/demo/Fact/Part0/Segment_0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sandeep purohit updated CARBONDATA-591:
---
Attachment: screenshot-1.png

> Remove unused code for spark 2.0 datatype utils
> ---
>
> Key: CARBONDATA-591
> URL: https://issues.apache.org/jira/browse/CARBONDATA-591
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: sandeep purohit
>Priority: Trivial
> Attachments: screenshot-1.png
>
>
> Remove unused code for data type utils for spark 2.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CARBONDATA-591) Remove unused code for spark 2.0 datatype utils

[sandeep purohit (JIRA)]

sandeep purohit created CARBONDATA-591:
--

 Summary: Remove unused code for spark 2.0 datatype utils
 Key: CARBONDATA-591
 URL: https://issues.apache.org/jira/browse/CARBONDATA-591
 Project: CarbonData
  Issue Type: Improvement
Reporter: sandeep purohit
Priority: Trivial


Remove unused code for data type utils for spark 2.0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
@jackylk I cannot squash the commits, can you squash the commits to single.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (CARBONDATA-583) Replace Function is not working for string/char

[Naresh P R (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798030#comment-15798030
 ] 

Naresh P R commented on CARBONDATA-583:
---

Hi, Can u try to use spark regexp_replace function instead of adding new udf.

eg., select regexp_replace('aaabbccaabb', 'aaa', 't');

which will return expected output :  "tbbccaabb"

> Replace Function is not working  for string/char
> 
>
> Key: CARBONDATA-583
> URL: https://issues.apache.org/jira/browse/CARBONDATA-583
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.0.0-incubating
> Environment: cluster
>Reporter: Anurag Srivastava
>Assignee: Rahul Kumar
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I am running "replace" function but it is giving error : "undefined function 
> replace".
> Query : select replace('aaabbccaabb', 'aaa', 't');
> Expected Result : "tbbccaabb"
> Result : Error: org.apache.spark.sql.AnalysisException: undefined function 
> replace; line 1 pos 30 (state=,code=0) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata issue #492: [CARBONDATA-440] Providing the update and d...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/492
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/441/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (CARBONDATA-589) carbon spark shell is not working with spark 2.0

[anubhav tarar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anubhav tarar updated CARBONDATA-589:
-
Description: 
carbon shell is not working with spark 2.0 version 
here are the logs

./carknoldus@knoldus:~/Desktop/open source/incubator-carbondata/bin$ 
./carbon-spark-shell
java.lang.ClassNotFoundException: org.apache.spark.repl.carbon.Main
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


  was:
carbon shell is not working with spark 2.0 version 
here are the logs

ava.lang.ClassNotFoundException: org.apache.spark.repl.carbon.Main
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



> carbon spark shell is not working with spark 2.0
> 
>
> Key: CARBONDATA-589
> URL: https://issues.apache.org/jira/browse/CARBONDATA-589
> Project: CarbonData
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.0.0-incubating
>Reporter: anubhav tarar
>Priority: Minor
>
> carbon shell is not working with spark 2.0 version 
> here are the logs
> ./carknoldus@knoldus:~/Desktop/open source/incubator-carbondata/bin$ 
> ./carbon-spark-shell
> java.lang.ClassNotFoundException: org.apache.spark.repl.carbon.Main
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:348)
>   at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
>   at 
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
>   at 
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
>   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
>   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
>   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata issue #492: [CARBONDATA-440] Providing the update and d...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/492
  
Build Failed  with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/439/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #495: [CARBONDATA-570] clean up carbon-hadoop mod...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/495
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/438/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/437/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #495: [CARBONDATA-570] clean up carbon-hadoop mod...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/495
  
Build Failed  with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/436/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #495: [CARBONDATA-570] clean up carbon-had...

[jackylk]

GitHub user jackylk opened a pull request:

https://github.com/apache/incubator-carbondata/pull/495

[CARBONDATA-570] clean up carbon-hadoop module

1. remove unused declaration
2. remove redundant exception

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata cleanhadoop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-carbondata/pull/495.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #495


commit 6b546ce804fe0f959e7c6997306df95d4bc7ff65
Author: jackylk 
Date:   2017-01-04T10:34:18Z

clean up hadoop




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/434/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (CARBONDATA-572) clean up code for carbon-spark-common module

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-572.
-
Resolution: Fixed
  Assignee: Jacky Li

> clean up code for carbon-spark-common module
> 
>
> Key: CARBONDATA-572
> URL: https://issues.apache.org/jira/browse/CARBONDATA-572
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Jacky Li
>Assignee: Jacky Li
> Fix For: 1.0.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/494


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/433/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/494#discussion_r94552530
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala
 ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
   def loadDataFrame(sqlContext: SQLContext,
   carbonLoadModel: CarbonLoadModel): DataFrame = {
-val df = sqlContext.read
-  .format("com.databricks.spark.csv.newapi")
-  .option("header", {
-if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
-  "true"
-} else {
-  "false"
-}
-  })
-  .option("delimiter", {
-if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
-  "" + DEFAULT_SEPARATOR
-} else {
-  carbonLoadModel.getCsvDelimiter
+  val hadoopConfiguration = new Configuration()
+  CommonUtil.configureCSVInputFormat(hadoopConfiguration, 
carbonLoadModel)
+  hadoopConfiguration.set(FileInputFormat.INPUT_DIR, 
carbonLoadModel.getFactFilePath)
--- End diff --

ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #484: [CARBONDATA-571][CARBONDATA-572] Cle...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-carbondata/pull/484


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #472: [CARBONDATA-568] clean up code for carbon-c...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/472
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/432/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #481: [WIP]reuse test case for integration module

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/incubator-carbondata/pull/481
  
Build Success with Spark 1.5.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/431/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata issue #484: [CARBONDATA-571][CARBONDATA-572] Clean up c...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/incubator-carbondata/pull/484
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...

[QiangCai]

Github user QiangCai commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/494#discussion_r94545504
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala
 ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
   def loadDataFrame(sqlContext: SQLContext,
   carbonLoadModel: CarbonLoadModel): DataFrame = {
-val df = sqlContext.read
-  .format("com.databricks.spark.csv.newapi")
-  .option("header", {
-if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
-  "true"
-} else {
-  "false"
-}
-  })
-  .option("delimiter", {
-if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
-  "" + DEFAULT_SEPARATOR
-} else {
-  carbonLoadModel.getCsvDelimiter
+  val hadoopConfiguration = new Configuration()
+  CommonUtil.configureCSVInputFormat(hadoopConfiguration, 
carbonLoadModel)
+  hadoopConfiguration.set(FileInputFormat.INPUT_DIR, 
carbonLoadModel.getFactFilePath)
--- End diff --

FileInputFormat.addInputPath method need a Job type paramter.
In addition, this FactFilePath already consist of all file path, we can 
directly set input path, no need to separate path and add path again. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-carbondata pull request #494: [CARBONDATA-218]Using CSVInputFormat...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/incubator-carbondata/pull/494#discussion_r94544073
  
--- Diff: 
integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/GlobalDictionaryUtil.scala
 ---
@@ -356,37 +363,49 @@ object GlobalDictionaryUtil {
*/
   def loadDataFrame(sqlContext: SQLContext,
   carbonLoadModel: CarbonLoadModel): DataFrame = {
-val df = sqlContext.read
-  .format("com.databricks.spark.csv.newapi")
-  .option("header", {
-if (StringUtils.isEmpty(carbonLoadModel.getCsvHeader)) {
-  "true"
-} else {
-  "false"
-}
-  })
-  .option("delimiter", {
-if (StringUtils.isEmpty(carbonLoadModel.getCsvDelimiter)) {
-  "" + DEFAULT_SEPARATOR
-} else {
-  carbonLoadModel.getCsvDelimiter
+  val hadoopConfiguration = new Configuration()
+  CommonUtil.configureCSVInputFormat(hadoopConfiguration, 
carbonLoadModel)
+  hadoopConfiguration.set(FileInputFormat.INPUT_DIR, 
carbonLoadModel.getFactFilePath)
--- End diff --

use `FileInputFormat.addInput`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (CARBONDATA-590) unusual behaviour of using carbonthrift server with spark 2.0

[anubhav tarar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anubhav tarar updated CARBONDATA-590:
-
Summary: unusual behaviour of using carbonthrift server with spark 2.0  
(was: unusual behaviour of carbonthrift server)

> unusual behaviour of using carbonthrift server with spark 2.0
> -
>
> Key: CARBONDATA-590
> URL: https://issues.apache.org/jira/browse/CARBONDATA-590
> Project: CarbonData
>  Issue Type: Bug
>  Components: build
>Affects Versions: 1.0.0-incubating
>Reporter: anubhav tarar
>
> have a look at these logs
> CREATE TABLE Bug212(int string)USING org.apache.spark.sql.CarbonSource
> OPTIONS("bucketnumber"="1", "bucketcolumns"="String","tableName"="t100");
> Error:
> org.apache.carbondata.spark.exception.MalformedCarbonCommandException: Table
> default.t 100 can not be created without key columns. Please use
> DICTIONARY_INCLUDE or DICTIONARY_EXCLUDE to set at least one key column if
> all specified columns are numeric types (state=,code=0)
> 2 minutes later
>  CREATE TABLE Bug211(int int)USING org.apache.spark.sql.CarbonSource
> OPTIONS("bucketnumber"="1", "bucketcolumns"="String","tableName"="t 100");
> CREATE TABLE Bug211(int int)USING org.apache.spark.sql.CarbonSource
> OPTIONS("bucketnumber"="1", "bucketcolumns"="String","tableName"="t 100");



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)