[GitHub] carbondata issue #1988: [CARBONDATA-2193] Support register analyzer and opti...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1988 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3698/ ---
[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1510 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3699/ ---
[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1959 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2684/ ---
[jira] [Updated] (CARBONDATA-2205) Unable to create a table with char data type with Spark 2.2.1
[ https://issues.apache.org/jira/browse/CARBONDATA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vandana Yadav updated CARBONDATA-2205: -- Description: Unable to create a table with char data type with Spark 2.2.1 Steps to reproduce: 1) create a table with char data type: CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by 'carbondata'; Expected Result: Table should be created successfully. Actual Result: Error: org.apache.spark.sql.AnalysisException: == Parse1 == DataType char is not supported.(line 1, pos 30) == SQL == CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by 'carbondata' --^^^ == Parse2 == [1.8] failure: identifier matching regex (?i)DATAMAP expected CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by 'carbondata' ^;; (state=,code=0) Note: Creating a table with char data type we don't need to provide any size with spark 2.1.0, but in spark 2.2.1 it's throwing a parsing error but when we provide size with char data type it lets us create a table successfully, we need to update this syntactical error in our documentation for reference. was: Unable to create a table with char data type with Spark 2.2.1 Steps to reproduce: 1) create a table with char data type: CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by 'carbondata'; Expected Result: Table should be created successfully. Actual Result: Error: org.apache.spark.sql.AnalysisException: == Parse1 == DataType char is not supported.(line 1, pos 30) == SQL == CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by 'carbondata' --^^^ == Parse2 == [1.8] failure: identifier matching regex (?i)DATAMAP expected CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by 'carbondata' ^;; (state=,code=0) > Unable to create a table with char data type with Spark 2.2.1 > - > > Key: CARBONDATA-2205 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2205 > Project: CarbonData > Issue Type: Bug > Components: data-query >Affects Versions: 1.4.0 >Reporter: Vandana Yadav >Priority: Minor > > Unable to create a table with char data type with Spark 2.2.1 > Steps to reproduce: > 1) create a table with char data type: > CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by > 'carbondata'; > Expected Result: Table should be created successfully. > Actual Result: > Error: org.apache.spark.sql.AnalysisException: == Parse1 == > DataType char is not supported.(line 1, pos 30) > == SQL == > CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by > 'carbondata' > --^^^ > == Parse2 == > [1.8] failure: identifier matching regex (?i)DATAMAP expected > CREATE TABLE uniqdata_char(id char) partitioned by (name char) stored by > 'carbondata' > ^;; (state=,code=0) > > Note: Creating a table with char data type we don't need to provide any size > with spark 2.1.0, but in spark 2.2.1 it's throwing a parsing error but when > we provide size with char data type it lets us create a table successfully, > we need to update this syntactical error in our documentation for reference. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1991: [CARBONDATA-2196]carbontable should be taken from lo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1991 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2685/ ---
[jira] [Resolved] (CARBONDATA-1114) Failed to run tests in windows env
[ https://issues.apache.org/jira/browse/CARBONDATA-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-1114. -- Resolution: Fixed Fix Version/s: 1.4.0 > Failed to run tests in windows env > -- > > Key: CARBONDATA-1114 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1114 > Project: CarbonData > Issue Type: Bug > Components: build >Affects Versions: 1.2.0 > Environment: windows7 64bit >Reporter: xuchuanyin >Priority: Minor > Labels: build, test > Fix For: 1.4.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > when run build command "mvn clean package -Pspark-2.1 -Dspark.version=2.1.0 > -Dmaven.test.failure.ignore=true -Dmaven.test.error.ignore=true" in windows > env, there are some failures in tests as below: > **MODULE:processing** > Failed tests: > LocalFileLockTest.testingLocalFileLockingByAcquiring2Locks:64 null > Tests in error: > ZooKeeperLockingTest.testZooKeeperLockingByTryingToAcquire2Locks:98 ? > StringIndexOutOfBounds > Tests run: 13, Failures: 1, Errors: 1, Skipped: 1 > **MODULE:core** > Failed tests: > AbsoluteTableIdentifierTest.gettablePathTest:89 null > DFSFileHolderImplUnitTest.testDouble:129 > Expected: is <7.3083355198552433E18> > but: was <8.0305922754531471E18> > DFSFileHolderImplUnitTest.testReadByteArray:84 > Expected: is [<72>] > but: was [<99>] > DFSFileHolderImplUnitTest.testReadByteArrayWithFilePath:90 > Expected: is [<108>, <108>] > but: was [<114>, <101>] > DFSFileHolderImplUnitTest.testReadInt:112 > Expected: is <1701604463> > but: was <1869767968> > DFSFileHolderImplUnitTest.testReadIntWithFileName:118 > Expected: is <1701604463> > but: was <1869767968> > DFSFileHolderImplUnitTest.testReadLong:96 > Expected: is <7308335519855243122L> > but: was <8030592275453146721L> > FileFactoryImplUnitTest.testCreateNewFileWithDefaultFileType:85 null > FileFactoryImplUnitTest.testCreateNewLockFileWithDefaultFileType:90 null > FileFactoryImplUnitTest.testCreateNewLockFileWithViewFsFileType:95 null > FileHolderImplUnitTest.testDouble:130 > Expected: is <7.3083355198552433E18> > but: was <8.0305922754531471E18> > FileHolderImplUnitTest.testReadByteArray:87 > Expected: is [<72>] > but: was [<99>] > FileHolderImplUnitTest.testReadByteArrayWithFilePath:93 > Expected: is [<108>, <108>] > but: was [<114>, <101>] > FileHolderImplUnitTest.testReadInt:113 > Expected: is <1701604463> > but: was <1869767968> > FileHolderImplUnitTest.testReadIntWithFileName:119 > Expected: is <1701604463> > but: was <1869767968> > FileHolderImplUnitTest.testReadLong:99 > Expected: is <7308335519855243122L> > but: was <8030592275453146721L> > LocalCarbonFileTest.testRenameForce:123 null > LocalCarbonFileTest.testsetLastModifiedTime:139 null > Tests run: 841, Failures: 18, Errors: 0, Skipped: 0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1988: [CARBONDATA-2193] Support register analyzer and opti...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1988 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3928/ ---
[jira] [Updated] (CARBONDATA-1114) Failed to run tests in windows env
[ https://issues.apache.org/jira/browse/CARBONDATA-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li updated CARBONDATA-1114: - Fix Version/s: 1.3.2 > Failed to run tests in windows env > -- > > Key: CARBONDATA-1114 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1114 > Project: CarbonData > Issue Type: Bug > Components: build >Affects Versions: 1.2.0 > Environment: windows7 64bit >Reporter: xuchuanyin >Priority: Minor > Labels: build, test > Fix For: 1.4.0, 1.3.2 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > when run build command "mvn clean package -Pspark-2.1 -Dspark.version=2.1.0 > -Dmaven.test.failure.ignore=true -Dmaven.test.error.ignore=true" in windows > env, there are some failures in tests as below: > **MODULE:processing** > Failed tests: > LocalFileLockTest.testingLocalFileLockingByAcquiring2Locks:64 null > Tests in error: > ZooKeeperLockingTest.testZooKeeperLockingByTryingToAcquire2Locks:98 ? > StringIndexOutOfBounds > Tests run: 13, Failures: 1, Errors: 1, Skipped: 1 > **MODULE:core** > Failed tests: > AbsoluteTableIdentifierTest.gettablePathTest:89 null > DFSFileHolderImplUnitTest.testDouble:129 > Expected: is <7.3083355198552433E18> > but: was <8.0305922754531471E18> > DFSFileHolderImplUnitTest.testReadByteArray:84 > Expected: is [<72>] > but: was [<99>] > DFSFileHolderImplUnitTest.testReadByteArrayWithFilePath:90 > Expected: is [<108>, <108>] > but: was [<114>, <101>] > DFSFileHolderImplUnitTest.testReadInt:112 > Expected: is <1701604463> > but: was <1869767968> > DFSFileHolderImplUnitTest.testReadIntWithFileName:118 > Expected: is <1701604463> > but: was <1869767968> > DFSFileHolderImplUnitTest.testReadLong:96 > Expected: is <7308335519855243122L> > but: was <8030592275453146721L> > FileFactoryImplUnitTest.testCreateNewFileWithDefaultFileType:85 null > FileFactoryImplUnitTest.testCreateNewLockFileWithDefaultFileType:90 null > FileFactoryImplUnitTest.testCreateNewLockFileWithViewFsFileType:95 null > FileHolderImplUnitTest.testDouble:130 > Expected: is <7.3083355198552433E18> > but: was <8.0305922754531471E18> > FileHolderImplUnitTest.testReadByteArray:87 > Expected: is [<72>] > but: was [<99>] > FileHolderImplUnitTest.testReadByteArrayWithFilePath:93 > Expected: is [<108>, <108>] > but: was [<114>, <101>] > FileHolderImplUnitTest.testReadInt:113 > Expected: is <1701604463> > but: was <1869767968> > FileHolderImplUnitTest.testReadIntWithFileName:119 > Expected: is <1701604463> > but: was <1869767968> > FileHolderImplUnitTest.testReadLong:99 > Expected: is <7308335519855243122L> > but: was <8030592275453146721L> > LocalCarbonFileTest.testRenameForce:123 null > LocalCarbonFileTest.testsetLastModifiedTime:139 null > Tests run: 841, Failures: 18, Errors: 0, Skipped: 0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2206) Integrate lucene as datamap
Jacky Li created CARBONDATA-2206: Summary: Integrate lucene as datamap Key: CARBONDATA-2206 URL: https://issues.apache.org/jira/browse/CARBONDATA-2206 Project: CarbonData Issue Type: Sub-task Reporter: Jacky Li Implement using lucene as one DataMap and adapt to DataMapFactory -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #1991: [CARBONDATA-2196]carbontable should be taken from lo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1991 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3929/ ---
[jira] [Assigned] (CARBONDATA-2206) Integrate lucene as datamap
[ https://issues.apache.org/jira/browse/CARBONDATA-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li reassigned CARBONDATA-2206: Assignee: Jacky Li > Integrate lucene as datamap > --- > > Key: CARBONDATA-2206 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2206 > Project: CarbonData > Issue Type: Sub-task >Reporter: Jacky Li >Assignee: Jacky Li >Priority: Major > > Implement using lucene as one DataMap and adapt to DataMapFactory -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/2003
[CARBONDATA-2206] support lucene index datamap
This PR is an initial effort to integrate lucene as an index datamap into
carbondata.
A new module called carbondata-lucene is added to support lucene datamap:
1. Add LuceneFineGrainDataMap, implement FineGrainDataMap interface.
2. Add LuceneCoarseGrainDataMap, implement CoarseGrainDataMap interface.
3. Support writing lucene index via LuceneDataMapWriter.
4. Implement LuceneDataMapFactory
5. A UDF called `text_match` is added
User can use lucene datamap as:
```
CREATE TABLE main(id INT, name STRING, city STRING, age INT)
STORED BY 'carbondata'
CREATE DATAMAP dm ON TABLE main
USING
'org.apache.carbondata.datamap.lucene.LuceneFineGrainDataMapFactory'
SELECT * FROM main WHERE TEXT_MATCH('name:n10')
```
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata
lucene-datamap-initial2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2003.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2003
commit e1d5b6c88b06d0c9d418008002d10a52368a0d84
Author: Jacky Li
Date: 2018-02-26T08:30:38Z
support lucene index datamap
---
[GitHub] carbondata issue #1982: [CARBONDATA-2184]Improve memory reuse for heap memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1982 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2686/ ---
[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1959 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3930/ ---
[GitHub] carbondata issue #1991: [CARBONDATA-2196]carbontable should be taken from lo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1991 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2687/ ---
[GitHub] carbondata issue #1982: [CARBONDATA-2184]Improve memory reuse for heap memor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1982 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3931/ ---
[GitHub] carbondata pull request #1867: [CARBONDATA-2055][Streaming][WIP]Support inte...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1867#discussion_r170854883 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSparkStreamFactory.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql + +import org.apache.commons.lang3.StringUtils + +import org.apache.carbondata.streaming.CarbonStreamException +import org.apache.carbondata.streaming.CarbonStreamSparkStreaming +import org.apache.carbondata.streaming.CarbonStreamSparkStreamingWriter + +/** + * Create CarbonStreamSparkStreamingWriter for stream table --- End diff -- change to `Create [[CarbonStreamSparkStreamingWriter]] for stream table` ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3933/ ---
[GitHub] carbondata pull request #1867: [CARBONDATA-2055][Streaming][WIP]Support inte...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1867#discussion_r170856116
--- Diff:
streaming/src/main/scala/org/apache/carbondata/streaming/CarbonStreamSparkStreaming.scala
---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.streaming
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.execution.streaming.CarbonAppendableStreamSink
+import org.apache.spark.sql.execution.streaming.Sink
+import org.apache.spark.sql.SaveMode
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.streaming.Time
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.locks.{CarbonLockFactory, ICarbonLock,
LockUsage}
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+
+class CarbonStreamSparkStreamingWriter {
+
+ private val LOGGER =
LogServiceFactory.getLogService(this.getClass.getCanonicalName)
+
+ private var isInitialize: Boolean = false
+
+ private var lock: ICarbonLock = null
+ private var carbonTable: CarbonTable = null
+ private var configuration: Configuration = null
+ private var carbonAppendableStreamSink: Sink = null
+ private val sparkSession: SparkSession =
SparkSession.builder().getOrCreate()
+
+ def this(carbonTable: CarbonTable, configuration: Configuration) {
+this()
+this.carbonTable = carbonTable
+this.configuration = configuration
+this.option("dbName", carbonTable.getDatabaseName)
+this.option("tableName", carbonTable.getTableName)
+ }
+
+ /**
+ * Acquired the lock for stream table
+ */
+ def lockStreamTable(): Unit = {
+lock =
CarbonLockFactory.getCarbonLockObj(carbonTable.getAbsoluteTableIdentifier,
+ LockUsage.STREAMING_LOCK)
+if (lock.lockWithRetries()) {
+ LOGGER.info("Acquired the lock for stream table: " +
+ carbonTable.getDatabaseName + "." +
+ carbonTable.getTableName)
+} else {
+ LOGGER.error("Not able to acquire the lock for stream table:" +
+ carbonTable.getDatabaseName + "." +
carbonTable.getTableName)
+ throw new InterruptedException(
+"Not able to acquire the lock for stream table: " +
carbonTable.getDatabaseName + "." +
+carbonTable.getTableName)
+}
+ }
+
+ /**
+ * unlock for stream table
+ */
+ def unLockStreamTable(): Unit = {
+if (null != lock) {
+ lock.unlock()
+ LOGGER.info("unlock for stream table: " +
+ carbonTable.getDatabaseName + "." +
+ carbonTable.getTableName)
+}
+ }
+
+ def initialize(): Unit = {
+carbonAppendableStreamSink = StreamSinkFactory.createStreamTableSink(
+ sparkSession,
+ configuration,
+ carbonTable,
+ extraOptions.toMap).asInstanceOf[CarbonAppendableStreamSink]
+
+lockStreamTable()
+
+isInitialize = true
+ }
+
+ def writeStreamData(dataFrame: DataFrame, time: Time): Unit = {
+if (!isInitialize) {
+ initialize()
+}
+carbonAppendableStreamSink.addBatch(time.milliseconds, dataFrame)
+ }
+
+ private val extraOptions = new scala.collection.mutable.HashMap[String,
String]
+ private var mode: SaveMode = SaveMode.ErrorIfExists
+
+ /**
+ * Specifies the behavior when data or table already exists. Options
include:
+ * - `SaveMode.Overwrite`: overwrite the existing data.
+ * - `SaveMode.Append`: append the data.
+ * - `SaveMode.Ignore`: ignore the operation (i.e. no-op).
+ * - `SaveMode.ErrorIfExists`: default option, throw an exception at
runtime.
+ */
+ def mode(
[GitHub] carbondata pull request #1867: [CARBONDATA-2055][Streaming][WIP]Support inte...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1867#discussion_r170856731
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSparkStreamFactory.scala
---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.commons.lang3.StringUtils
+
+import org.apache.carbondata.streaming.CarbonStreamException
+import org.apache.carbondata.streaming.CarbonStreamSparkStreaming
+import org.apache.carbondata.streaming.CarbonStreamSparkStreamingWriter
+
+/**
+ * Create CarbonStreamSparkStreamingWriter for stream table
+ * when integrate with Spark Streaming
+ */
+object CarbonSparkStreamFactory {
+
+ def getStreamSparkStreamWriter(
+dbNameStr: String,
+tableName: String): CarbonStreamSparkStreamingWriter =
+synchronized {
+val dbName = if (StringUtils.isEmpty(dbNameStr)) "default" else
dbNameStr
+val key = dbName + "." + tableName
+if (CarbonStreamSparkStreaming.getTableMap.containsKey(key)) {
+ CarbonStreamSparkStreaming.getTableMap.get(key)
+} else {
+ if (StringUtils.isEmpty(tableName) || tableName.contains(" ")) {
+throw new CarbonStreamException("Table creation failed. " +
+"Table name must not be blank or "
+
+"cannot contain blank space")
+ }
+ val carbonTable = CarbonEnv.getCarbonTable(Some(dbName),
+tableName)(SparkSession.builder().getOrCreate())
--- End diff --
build two SparkSession repeatedly in line 47 and line 53, build one instead
---
[GitHub] carbondata issue #2002: [HOTFIX] Check concurrent loading in compaction comm...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2002 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3700/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1995 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3701/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2688/ ---
[GitHub] carbondata issue #1991: [CARBONDATA-2196]carbontable should be taken from lo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1991 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3932/ ---
[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1959 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2689/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2691/ ---
[GitHub] carbondata issue #1999: [CARBONDATA-2204] Optimized number of reads of table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1999 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3934/ ---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170865464
--- Diff:
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneCoarseGrainDataMapFactory.java
---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.lucene;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datamap.DataMapDistributable;
+import org.apache.carbondata.core.datamap.DataMapLevel;
+import org.apache.carbondata.core.datamap.dev.DataMapModel;
+import org.apache.carbondata.core.datamap.dev.cgdatamap.CoarseGrainDataMap;
+import org.apache.carbondata.core.memory.MemoryException;
+
+@InterfaceAudience.Internal
+public class LuceneCoarseGrainDataMapFactory extends
LuceneDataMapFactoryBase {
+ private static final LogService LOGGER =
+
LogServiceFactory.getLogService(LuceneCoarseGrainDataMapFactory.class.getName());
+
+ /**
+ * Get the datamap for segmentid
+ */
+ public List getDataMaps(String segmentId) throws
IOException {
+List lstDataMap = new ArrayList<>();
+CoarseGrainDataMap dataMap = new LuceneCoarseGrainDataMap(analyzer);
+try {
+ dataMap.init(new DataMapModel(
+ tableIdentifier.getTablePath() + "/Fact/Part0/Segment_" +
segmentId + File.separator
--- End diff --
please invoke getSegmentDIr
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170863724
--- Diff: datamap/lucene/pom.xml ---
@@ -0,0 +1,152 @@
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+
+org.apache.carbondata
+carbondata-parent
+1.4.0-SNAPSHOT
+../../pom.xml
+
+
+ carbondata-lucene
+ Apache CarbonData :: Lucene Index DataMap
+
+
+${basedir}/../../dev
+6.3.0
+6.3.0
+
+
+
+
+ org.apache.carbondata
+ carbondata-spark2
+ ${project.version}
+
+
+ org.apache.lucene
+ lucene-core
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.lucene
+ lucene-analyzers-common
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.lucene
+ lucene-queryparser
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.lucene
+ lucene-sandbox
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.solr
+ solr-core
+ ${solr.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.solr
+ solr-solrj
+ ${solr.version}
+
+
+ *
+ *
+
+
+
+
+ org.scalatest
+ scalatest_${scala.binary.version}
+ test
+
+
+
+
+src/test/scala
+
+
+src/resources
+
+
+.
+
+ CARBON_SPARK_INTERFACELogResource.properties
--- End diff --
not require
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170864075
--- Diff:
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/FilterExpressParser.java
---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.lucene;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.core.scan.filter.intf.ExpressionType;
+import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.queryparser.classic.QueryParser;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.Query;
+
+@InterfaceAudience.Internal
+public class FilterExpressParser extends QueryParser {
--- End diff --
remove unuseful class
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user QiangCai commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2003#discussion_r170863075 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/intf/ExpressionType.java --- @@ -42,5 +42,6 @@ TRUE, STARTSWITH, ENDSWITH, - CONTAINSWITH + CONTAINSWITH, + MATCH --- End diff -- please use TEXT_MATCH ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2692/ ---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170864420
--- Diff:
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneCoarseGrainDataMap.java
---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.lucene;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datamap.dev.DataMapModel;
+import org.apache.carbondata.core.datamap.dev.cgdatamap.CoarseGrainDataMap;
+import org.apache.carbondata.core.datastore.block.SegmentProperties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.indexstore.Blocklet;
+import org.apache.carbondata.core.memory.MemoryException;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.queryparser.classic.MultiFieldQueryParser;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.queryparser.classic.QueryParser;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.solr.store.hdfs.HdfsDirectory;
+
+@InterfaceAudience.Internal
+public class LuceneCoarseGrainDataMap extends CoarseGrainDataMap {
+
+ /**
+ * log information
+ */
+ private static final LogService LOGGER =
+
LogServiceFactory.getLogService(LuceneCoarseGrainDataMap.class.getName());
+
+ public static final int BLOCKID_ID = 0;
+
+ public static final int BLOCKLETID_ID = 1;
+
+ public static final int PAGEID_ID = 2;
+
+ public static final int ROWID_ID = 3;
+ /**
+ * searcher object for this datamap
+ */
+ private IndexSearcher indexSearcher = null;
+
+ /**
+ * default max values to return
+ */
+ private static int MAX_RESULT_NUMBER = 100;
--- End diff --
better to config it
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2003#discussion_r170865828 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/intf/ExpressionType.java --- @@ -42,5 +42,6 @@ TRUE, STARTSWITH, ENDSWITH, - CONTAINSWITH + CONTAINSWITH, + MATCH --- End diff -- fixed ---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170866003
--- Diff:
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/FilterExpressParser.java
---
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.lucene;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.core.scan.filter.intf.ExpressionType;
+import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.queryparser.classic.QueryParser;
+import org.apache.lucene.search.BooleanClause;
+import org.apache.lucene.search.Query;
+
+@InterfaceAudience.Internal
+public class FilterExpressParser extends QueryParser {
--- End diff --
fixed
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170865978
--- Diff: datamap/lucene/pom.xml ---
@@ -0,0 +1,152 @@
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ 4.0.0
+
+
+org.apache.carbondata
+carbondata-parent
+1.4.0-SNAPSHOT
+../../pom.xml
+
+
+ carbondata-lucene
+ Apache CarbonData :: Lucene Index DataMap
+
+
+${basedir}/../../dev
+6.3.0
+6.3.0
+
+
+
+
+ org.apache.carbondata
+ carbondata-spark2
+ ${project.version}
+
+
+ org.apache.lucene
+ lucene-core
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.lucene
+ lucene-analyzers-common
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.lucene
+ lucene-queryparser
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.lucene
+ lucene-sandbox
+ ${lucene.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.solr
+ solr-core
+ ${solr.version}
+
+
+ *
+ *
+
+
+
+
+ org.apache.solr
+ solr-solrj
+ ${solr.version}
+
+
+ *
+ *
+
+
+
+
+ org.scalatest
+ scalatest_${scala.binary.version}
+ test
+
+
+
+
+src/test/scala
+
+
+src/resources
+
+
+.
+
+ CARBON_SPARK_INTERFACELogResource.properties
--- End diff --
fixed
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170866114
--- Diff:
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneCoarseGrainDataMapFactory.java
---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.lucene;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datamap.DataMapDistributable;
+import org.apache.carbondata.core.datamap.DataMapLevel;
+import org.apache.carbondata.core.datamap.dev.DataMapModel;
+import org.apache.carbondata.core.datamap.dev.cgdatamap.CoarseGrainDataMap;
+import org.apache.carbondata.core.memory.MemoryException;
+
+@InterfaceAudience.Internal
+public class LuceneCoarseGrainDataMapFactory extends
LuceneDataMapFactoryBase {
+ private static final LogService LOGGER =
+
LogServiceFactory.getLogService(LuceneCoarseGrainDataMapFactory.class.getName());
+
+ /**
+ * Get the datamap for segmentid
+ */
+ public List getDataMaps(String segmentId) throws
IOException {
+List lstDataMap = new ArrayList<>();
+CoarseGrainDataMap dataMap = new LuceneCoarseGrainDataMap(analyzer);
+try {
+ dataMap.init(new DataMapModel(
+ tableIdentifier.getTablePath() + "/Fact/Part0/Segment_" +
segmentId + File.separator
--- End diff --
fixed
---
[GitHub] carbondata pull request #2003: [CARBONDATA-2206] support lucene index datama...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2003#discussion_r170866077
--- Diff:
datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneCoarseGrainDataMap.java
---
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.lucene;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datamap.dev.DataMapModel;
+import org.apache.carbondata.core.datamap.dev.cgdatamap.CoarseGrainDataMap;
+import org.apache.carbondata.core.datastore.block.SegmentProperties;
+import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.indexstore.Blocklet;
+import org.apache.carbondata.core.memory.MemoryException;
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier;
+import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.queryparser.classic.MultiFieldQueryParser;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.queryparser.classic.QueryParser;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.solr.store.hdfs.HdfsDirectory;
+
+@InterfaceAudience.Internal
+public class LuceneCoarseGrainDataMap extends CoarseGrainDataMap {
+
+ /**
+ * log information
+ */
+ private static final LogService LOGGER =
+
LogServiceFactory.getLogService(LuceneCoarseGrainDataMap.class.getName());
+
+ public static final int BLOCKID_ID = 0;
+
+ public static final int BLOCKLETID_ID = 1;
+
+ public static final int PAGEID_ID = 2;
+
+ public static final int ROWID_ID = 3;
+ /**
+ * searcher object for this datamap
+ */
+ private IndexSearcher indexSearcher = null;
+
+ /**
+ * default max values to return
+ */
+ private static int MAX_RESULT_NUMBER = 100;
--- End diff --
I will leave it to future PR
---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3937/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2693/ ---
[GitHub] carbondata issue #1999: [CARBONDATA-2204] Optimized number of reads of table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1999 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2690/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3938/ ---
[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1959 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3935/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2694/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2695/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3939/ ---
[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1959 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3702/ ---
[jira] [Created] (CARBONDATA-2207) TestCase Fails using Hive Metastore
Jatin created CARBONDATA-2207: - Summary: TestCase Fails using Hive Metastore Key: CARBONDATA-2207 URL: https://issues.apache.org/jira/browse/CARBONDATA-2207 Project: CarbonData Issue Type: Bug Affects Versions: 1.4.0 Reporter: Jatin Assignee: Jatin Fix For: 1.4.0 Run All the Cabon TestCases using hive metastore out of which some test cases were failing because of not getting carbon table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2208) Pre aggregate datamap creation is failing when count(*) present in query
kumar vishal created CARBONDATA-2208: Summary: Pre aggregate datamap creation is failing when count(*) present in query Key: CARBONDATA-2208 URL: https://issues.apache.org/jira/browse/CARBONDATA-2208 Project: CarbonData Issue Type: Bug Reporter: kumar vishal Assignee: kumar vishal Pre aggregate data map creation is failing with parsing error create datamap agg9 on table maintable using 'preaggregate' as select name, count(*) from maintable group by name -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
GitHub user kumarvishal09 opened a pull request: https://github.com/apache/carbondata/pull/2004 [CARBONDATA-2208]Pre aggregate datamap creation is failing when count(*) present in query Pre aggregate data map creation is failing with parsing error create datamap agg9 on table maintable using 'preaggregate' as select name, count from maintable group by name your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Added UT - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kumarvishal09/incubator-carbondata master_26-02-2018 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2004.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2004 commit 7b51a59148b570093fe82f3a14217c1c24886509 Author: kumarvishal Date: 2018-02-27T10:31:06Z Fixed count(*) query issue in pre aggregate ---
[GitHub] carbondata issue #1982: [CARBONDATA-2184]Improve memory reuse for heap memor...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/1982 retest sdv please. ---
[GitHub] carbondata pull request #2005: [CARBONDATA-2207] Fix testcases after using h...
GitHub user jatin9896 opened a pull request: https://github.com/apache/carbondata/pull/2005 [CARBONDATA-2207] Fix testcases after using hive metastore CarbonTable was getting null in case of hivemetatore so, fetch the same from metastore instead of carbon. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed?No - [ ] Any backward compatibility impacted?No - [ ] Document update required?No - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jatin9896/incubator-carbondata CARBONDATA-2207 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2005.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2005 commit 50fadb13b63822a7c0bca177a01b6a09e62136d6 Author: Jatin Date: 2018-02-27T10:43:40Z fix for hivemetastore using for carbon testcases ---
[jira] [Updated] (CARBONDATA-2208) Pre aggregate datamap creation is failing when count(*) present in query
[ https://issues.apache.org/jira/browse/CARBONDATA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal updated CARBONDATA-2208: - Description: Pre aggregate data map creation is failing with parsing error create datamap agg on table maintable using 'preaggregate' as select name, count(*) from maintable group by name was: Pre aggregate data map creation is failing with parsing error create datamap agg9 on table maintable using 'preaggregate' as select name, count(*) from maintable group by name > Pre aggregate datamap creation is failing when count(*) present in query > > > Key: CARBONDATA-2208 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2208 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Pre aggregate data map creation is failing with parsing error > create datamap agg on table maintable using 'preaggregate' as select name, > count(*) from maintable group by name > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2006: Fixed rename table with partitions not workin...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2006 Fixed rename table with partitions not working issue and batch_sort and no_sort with partition table issue Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata partition-batchsort-issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2006.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2006 commit 5f60c0a312a9670f46dccff5c77b9e271a8a8ee4 Author: ravipesala Date: 2018-02-27T11:08:09Z Fixed rename table with partitions not working issue and batch_sort and no_sort with partition table issue ---
[jira] [Created] (CARBONDATA-2209) Rename table with partitions not working issue and batch_sort and no_sort with partition table issue
Ravindra Pesala created CARBONDATA-2209: --- Summary: Rename table with partitions not working issue and batch_sort and no_sort with partition table issue Key: CARBONDATA-2209 URL: https://issues.apache.org/jira/browse/CARBONDATA-2209 Project: CarbonData Issue Type: Bug Reporter: Ravindra Pesala # After table rename on partitions table, it returns empty data upon querying. # Batch sort and no sort loading is not working on partition table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2004: [CARBONDATA-2208]Pre aggregate datamap creation is f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2004 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3940/ ---
[GitHub] carbondata issue #2004: [CARBONDATA-2208]Pre aggregate datamap creation is f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2004 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2696/ ---
[GitHub] carbondata issue #2005: [CARBONDATA-2207] Fix testcases after using hive met...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2005 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2697/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2698/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3944/ ---
[GitHub] carbondata issue #2005: [CARBONDATA-2207] Fix testcases after using hive met...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2005 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3942/ ---
[GitHub] carbondata issue #2006: [CARBONDATA-2209] Fixed rename table with partitions...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2006 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2699/ ---
[jira] [Created] (CARBONDATA-2210) Not able to rename a partitioned table on cluster
Vandana Yadav created CARBONDATA-2210:
-
Summary: Not able to rename a partitioned table on cluster
Key: CARBONDATA-2210
URL: https://issues.apache.org/jira/browse/CARBONDATA-2210
Project: CarbonData
Issue Type: Bug
Components: data-query
Affects Versions: 1.4.0
Environment: spark 2.2.1
Reporter: Vandana Yadav
Not able to rename a partitioned table on cluster
Steps to reproduce:
1) Create a hive table:
CREATE TABLE uniqdata_hive (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double, INTEGER_COLUMN1
int)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOAD DATA LOCAL INPATH
'/opt/Carbon/CarbonData/TestData/Data/uniqdata/2000_UniqData.csv' into table
UNIQDATA_HIVE
2) Create a carbon table:
CREATE TABLE uniqdata_int (CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB
timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double, INTEGER_COLUMN1
int) Partitioned by (cust_id int) STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES ("TABLE_BLOCKSIZE"= "256 MB")
insert into uniqdata_int partition(cust_id)select * from uniqdata_hive limit 2
3) Execute Query:
alter table uniqdata_int rename to uniqdata_bigint1
Expected Result: table should be renamed successfully.
Actual Result:
operation failed for default.uniqdata_int: Alter table rename table operation
failed: Folder rename failed for table default.uniqdata_int
logs
[exec] 18/02/27 16:38:16 INFO SelectQuery: Executing Query: alter table
uniqdata_int rename to uniqdata_bigint1
[exec] 18/02/27 16:38:16 INFO CarbonSparkSqlParser: Parsing command: alter
table uniqdata_int rename to uniqdata_bigint1
[exec] 18/02/27 16:38:16 INFO CarbonLateDecodeRule: main skip
CarbonOptimizer
[exec] 18/02/27 16:38:16 INFO CarbonLateDecodeRule: main Skip
CarbonOptimizer
[exec] 18/02/27 16:38:16 INFO HiveMetaStore: 0: get_table : db=default
tbl=uniqdata_int
[exec] 18/02/27 16:38:16 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_table : db=default tbl=uniqdata_int
[exec] 18/02/27 16:38:16 INFO CatalystSqlParser: Parsing command: int
[exec] 18/02/27 16:38:16 INFO CatalystSqlParser: Parsing command:
array
[exec] 18/02/27 16:38:16 INFO HiveMetaStore: 0: get_table : db=default
tbl=uniqdata_bigint1
[exec] 18/02/27 16:38:16 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_table : db=default tbl=uniqdata_bigint1
[exec] 18/02/27 16:38:16 AUDIT CarbonAlterTableRenameCommand:
[hadoop-master][root][Thread-1]Rename table request has been received for
default.uniqdata_int
[exec] 18/02/27 16:38:16 INFO CarbonAlterTableRenameCommand: main Rename
table request has been received for default.uniqdata_int
[exec] 18/02/27 16:38:16 INFO HiveMetaStore: 0: get_table : db=default
tbl=uniqdata_int
[exec] 18/02/27 16:38:16 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_table : db=default tbl=uniqdata_int
[exec] 18/02/27 16:38:16 INFO CatalystSqlParser: Parsing command: int
[exec] 18/02/27 16:38:16 INFO CatalystSqlParser: Parsing command:
array
[exec] 18/02/27 16:38:16 INFO HiveMetaStore: 0: get_table : db=default
tbl=uniqdata_int
[exec] 18/02/27 16:38:16 INFO audit: ugi=root ip=unknown-ip-addr
cmd=get_table : db=default tbl=uniqdata_int
[exec] 18/02/27 16:38:16 INFO CatalystSqlParser: Parsing command: int
[exec] 18/02/27 16:38:16 INFO CatalystSqlParser: Parsing command:
array
[exec] 18/02/27 16:38:16 INFO HdfsFileLock: main HDFS lock
path:hdfs://hadoop-master:54311/opt/CarbonStore/default/uniqdata_int/meta.lock
[exec] 18/02/27 16:38:16 INFO CarbonLockUtil: main Trying to acquire lock:
meta.lockfor table: default_uniqdata_int
[exec] 18/02/27 16:38:16 INFO CarbonLockUtil: main Successfully acquired
the lock meta.lockfor table: default_uniqdata_int
[exec] 18/02/27 16:38:16 INFO HdfsFileLock: main HDFS lock
path:hdfs://hadoop-master:54311/opt/CarbonStore/default/uniqdata_int/compaction.lock
[exec] 18/02/27 16:38:16 INFO CarbonLockUtil: main Trying to acquire lock:
compaction.lockfor table: default_uniqdata_int
[exec] 18/02/27 16:38:16 INFO CarbonLockUtil: main Successfully acquired
the lock compaction.lockfor table: default_uniqdata_int
[exec] 18/02/27 16:38:16 INFO HdfsFileLock: main HDFS lock
path:hdfs://hadoop-master:54311/opt/CarbonStore/default/uniqdata_int/delete_segment.lock
[exec] 18/02/27 16:38:16 INFO CarbonLockUtil: main Trying to acquire lock:
delete_segment.lockfor table: default_uniqdata_int
[exec] 18/02/27 16:38:
[GitHub] carbondata issue #1991: [CARBONDATA-2196]carbontable should be taken from lo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1991 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3703/ ---
[GitHub] carbondata pull request #1988: [CARBONDATA-2193] Support register analyzer a...
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1988#discussion_r170894492
--- Diff:
integration/spark2/src/main/java/org/apache/carbondata/datamap/preaggregate/PreaggregateMVDataMapRules.java
---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.datamap.preaggregate;
+
+import org.apache.carbondata.datamap.MVDataMapRules;
+
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
+import org.apache.spark.sql.catalyst.rules.Rule;
+import org.apache.spark.sql.hive.CarbonPreAggregateDataLoadingRules;
+import org.apache.spark.sql.hive.CarbonPreAggregateQueryRules;
+
+public class PreaggregateMVDataMapRules implements MVDataMapRules {
--- End diff --
add interface annotation
---
[GitHub] carbondata issue #2006: [CARBONDATA-2209] Fixed rename table with partitions...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2006 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3943/ ---
[GitHub] carbondata issue #2005: [CARBONDATA-2207] Fix testcases after using hive met...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2005 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2701/ ---
[GitHub] carbondata issue #2006: [CARBONDATA-2209] Fixed rename table with partitions...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2006 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2700/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3947/ ---
[GitHub] carbondata issue #2006: [CARBONDATA-2209] Fixed rename table with partitions...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2006 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3945/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3949/ ---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170902884
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ---
@@ -75,7 +75,12 @@ public CarbonTablePath(CarbonTableIdentifier
carbonTableIdentifier, String table
* @param carbonFilePath
*/
public static String getFolderContainingFile(String carbonFilePath) {
-return carbonFilePath.substring(0, carbonFilePath.lastIndexOf('/'));
+int lastIndex = carbonFilePath.lastIndexOf('/');
+// below code for handling windows environment
+if (-1 == lastIndex) {
--- End diff --
Whether it need support this scenario: E:/xubo\idea ?
---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170906653
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -346,11 +346,13 @@ object PreAggregateUtil {
carbonTable)
}
// if parent column relation is of size more than one that means
aggregate table
-// column is derived from multiple column of main table
-// or if expression is not a instance of attribute reference
+// column is derived from multiple column of main table or if size is
zero then it means
+// column is present in select statement is some constants for example
count(*)
+// and if expression is not a instance of attribute reference
// then use column name which is passed
val columnName =
-if (parentColumnsName.size > 1 &&
!expression.isInstanceOf[AttributeReference]) {
+if ((parentColumnsName.size > 1 || parentColumnsName.isEmpty) &&
+!expression.isInstanceOf[AttributeReference]) {
newColumnName
--- End diff --
Why don't we use 1_count instead of 0_count in default?
It is count(1) after spark parser count(*)
---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170908712
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -346,11 +346,13 @@ object PreAggregateUtil {
carbonTable)
}
// if parent column relation is of size more than one that means
aggregate table
-// column is derived from multiple column of main table
-// or if expression is not a instance of attribute reference
+// column is derived from multiple column of main table or if size is
zero then it means
+// column is present in select statement is some constants for example
count(*)
+// and if expression is not a instance of attribute reference
// then use column name which is passed
val columnName =
-if (parentColumnsName.size > 1 &&
!expression.isInstanceOf[AttributeReference]) {
+if ((parentColumnsName.size > 1 || parentColumnsName.isEmpty) &&
+!expression.isInstanceOf[AttributeReference]) {
newColumnName
--- End diff --
i didn't get your comment
---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2702/ ---
[GitHub] carbondata issue #2005: [CARBONDATA-2207] Fix testcases after using hive met...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2005 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2703/ ---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170909437
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ---
@@ -75,7 +75,12 @@ public CarbonTablePath(CarbonTableIdentifier
carbonTableIdentifier, String table
* @param carbonFilePath
*/
public static String getFolderContainingFile(String carbonFilePath) {
-return carbonFilePath.substring(0, carbonFilePath.lastIndexOf('/'));
+int lastIndex = carbonFilePath.lastIndexOf('/');
+// below code for handling windows environment
+if (-1 == lastIndex) {
--- End diff --
in windows file seperator is "\\" , this is to handle the same
---
[GitHub] carbondata issue #2004: [CARBONDATA-2208]Pre aggregate datamap creation is f...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2004 LGTM, verified it. ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2706/ ---
[GitHub] carbondata issue #2005: [CARBONDATA-2207] Fix testcases after using hive met...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2005 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3948/ ---
[GitHub] carbondata issue #1993: [CARBONDATA-2199] Fixed Dimension column after restr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1993 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3950/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3951/ ---
[GitHub] carbondata issue #1982: [CARBONDATA-2184]Improve memory reuse for heap memor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1982 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3704/ ---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170916327
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ---
@@ -75,7 +75,12 @@ public CarbonTablePath(CarbonTableIdentifier
carbonTableIdentifier, String table
* @param carbonFilePath
*/
public static String getFolderContainingFile(String carbonFilePath) {
-return carbonFilePath.substring(0, carbonFilePath.lastIndexOf('/'));
+int lastIndex = carbonFilePath.lastIndexOf('/');
+// below code for handling windows environment
+if (-1 == lastIndex) {
--- End diff --
in windows, you code will return E: if you input E:/xubo\idea , not return
E:/xubo
---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170916613
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ---
@@ -75,7 +75,12 @@ public CarbonTablePath(CarbonTableIdentifier
carbonTableIdentifier, String table
* @param carbonFilePath
*/
public static String getFolderContainingFile(String carbonFilePath) {
-return carbonFilePath.substring(0, carbonFilePath.lastIndexOf('/'));
+int lastIndex = carbonFilePath.lastIndexOf('/');
+// below code for handling windows environment
+if (-1 == lastIndex) {
--- End diff --
E:/xubo\idea will never come either it will be E:/xubo/idea or
E:\xubo\idea ?
---
[GitHub] carbondata issue #1993: [CARBONDATA-2199] Fixed Dimension column after restr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1993 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2705/ ---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170919785
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala
---
@@ -346,11 +346,13 @@ object PreAggregateUtil {
carbonTable)
}
// if parent column relation is of size more than one that means
aggregate table
-// column is derived from multiple column of main table
-// or if expression is not a instance of attribute reference
+// column is derived from multiple column of main table or if size is
zero then it means
+// column is present in select statement is some constants for example
count(*)
+// and if expression is not a instance of attribute reference
// then use column name which is passed
val columnName =
-if (parentColumnsName.size > 1 &&
!expression.isInstanceOf[AttributeReference]) {
+if ((parentColumnsName.size > 1 || parentColumnsName.isEmpty) &&
+!expression.isInstanceOf[AttributeReference]) {
newColumnName
--- End diff --
ok, it understand. It's fine, thanks.
---
[GitHub] carbondata pull request #2004: [CARBONDATA-2208]Pre aggregate datamap creati...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2004#discussion_r170922780
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/CarbonTablePath.java ---
@@ -75,7 +75,12 @@ public CarbonTablePath(CarbonTableIdentifier
carbonTableIdentifier, String table
* @param carbonFilePath
*/
public static String getFolderContainingFile(String carbonFilePath) {
-return carbonFilePath.substring(0, carbonFilePath.lastIndexOf('/'));
+int lastIndex = carbonFilePath.lastIndexOf('/');
+// below code for handling windows environment
+if (-1 == lastIndex) {
--- End diff --
ok, it's fine
---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3952/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3953/ ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2707/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2708/ ---
[GitHub] carbondata issue #1993: [CARBONDATA-2199] Fixed Dimension column after restr...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1993 LGTM ---
[GitHub] carbondata issue #1991: [CARBONDATA-2196]carbontable should be taken from lo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1991 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3705/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2709/ ---
[GitHub] carbondata issue #1995: [WIP] File Format Reader
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1995 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3954/ ---
[GitHub] carbondata pull request #2007: [HOTFIX] Fix timestamp format in testcase
GitHub user jackylk opened a pull request: https://github.com/apache/carbondata/pull/2007 [HOTFIX] Fix timestamp format in testcase In UT testcases, there is random failure because of thread local variable in DataTypeUtil. This PR corrected it. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata carbonstore-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2007.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2007 commit 4be2f603d776027768676f8f47680be5e5d0a219 Author: Jacky Li Date: 2018-02-27T14:53:14Z fix ---
[GitHub] carbondata issue #2003: [CARBONDATA-2206] support lucene index datamap
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2003 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3955/ ---
[GitHub] carbondata pull request #1867: [CARBONDATA-2055][Streaming][WIP]Support inte...
Github user zzcclp commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1867#discussion_r170958463 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSparkStreamFactory.scala --- @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql + +import org.apache.commons.lang3.StringUtils + +import org.apache.carbondata.streaming.CarbonStreamException +import org.apache.carbondata.streaming.CarbonStreamSparkStreaming +import org.apache.carbondata.streaming.CarbonStreamSparkStreamingWriter + +/** + * Create CarbonStreamSparkStreamingWriter for stream table --- End diff -- Done ---
[GitHub] carbondata pull request #1867: [CARBONDATA-2055][Streaming][WIP]Support inte...
Github user zzcclp commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1867#discussion_r170958496
--- Diff:
streaming/src/main/scala/org/apache/carbondata/streaming/CarbonStreamSparkStreaming.scala
---
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.streaming
+
+import java.util
+
+import scala.collection.JavaConverters._
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.execution.streaming.CarbonAppendableStreamSink
+import org.apache.spark.sql.execution.streaming.Sink
+import org.apache.spark.sql.SaveMode
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.streaming.Time
+
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.locks.{CarbonLockFactory, ICarbonLock,
LockUsage}
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+
+class CarbonStreamSparkStreamingWriter {
+
+ private val LOGGER =
LogServiceFactory.getLogService(this.getClass.getCanonicalName)
+
+ private var isInitialize: Boolean = false
+
+ private var lock: ICarbonLock = null
+ private var carbonTable: CarbonTable = null
+ private var configuration: Configuration = null
+ private var carbonAppendableStreamSink: Sink = null
+ private val sparkSession: SparkSession =
SparkSession.builder().getOrCreate()
+
+ def this(carbonTable: CarbonTable, configuration: Configuration) {
+this()
+this.carbonTable = carbonTable
+this.configuration = configuration
+this.option("dbName", carbonTable.getDatabaseName)
+this.option("tableName", carbonTable.getTableName)
+ }
+
+ /**
+ * Acquired the lock for stream table
+ */
+ def lockStreamTable(): Unit = {
+lock =
CarbonLockFactory.getCarbonLockObj(carbonTable.getAbsoluteTableIdentifier,
+ LockUsage.STREAMING_LOCK)
+if (lock.lockWithRetries()) {
+ LOGGER.info("Acquired the lock for stream table: " +
+ carbonTable.getDatabaseName + "." +
+ carbonTable.getTableName)
+} else {
+ LOGGER.error("Not able to acquire the lock for stream table:" +
+ carbonTable.getDatabaseName + "." +
carbonTable.getTableName)
+ throw new InterruptedException(
+"Not able to acquire the lock for stream table: " +
carbonTable.getDatabaseName + "." +
+carbonTable.getTableName)
+}
+ }
+
+ /**
+ * unlock for stream table
+ */
+ def unLockStreamTable(): Unit = {
+if (null != lock) {
+ lock.unlock()
+ LOGGER.info("unlock for stream table: " +
+ carbonTable.getDatabaseName + "." +
+ carbonTable.getTableName)
+}
+ }
+
+ def initialize(): Unit = {
+carbonAppendableStreamSink = StreamSinkFactory.createStreamTableSink(
+ sparkSession,
+ configuration,
+ carbonTable,
+ extraOptions.toMap).asInstanceOf[CarbonAppendableStreamSink]
+
+lockStreamTable()
+
+isInitialize = true
+ }
+
+ def writeStreamData(dataFrame: DataFrame, time: Time): Unit = {
+if (!isInitialize) {
+ initialize()
+}
+carbonAppendableStreamSink.addBatch(time.milliseconds, dataFrame)
+ }
+
+ private val extraOptions = new scala.collection.mutable.HashMap[String,
String]
+ private var mode: SaveMode = SaveMode.ErrorIfExists
+
+ /**
+ * Specifies the behavior when data or table already exists. Options
include:
+ * - `SaveMode.Overwrite`: overwrite the existing data.
+ * - `SaveMode.Append`: append the data.
+ * - `SaveMode.Ignore`: ignore the operation (i.e. no-op).
+ * - `SaveMode.ErrorIfExists`: default option, throw an exception at
runtime.
+ */
+ def mode(s
