[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2190


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183274384
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) 
{
 return this;
   }
 
+  /**
+   * To support the load options for sdk writer
+   * @param options key,value pair of load options.
+   *supported keys values are
+   *a. bad_records_logger_enable -- true, false
--- End diff --

ok


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183242493
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -75,6 +78,11 @@ public TableSchemaBuilder tableName(String tableName) {
 return this;
   }
 
+  public TableSchemaBuilder resetTransactionalTable(boolean 
isTransactionalTable) {
--- End diff --

ok


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183239308
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -210,11 +260,26 @@ private CarbonTable buildCarbonTable() {
   tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize);
 }
 
-List sortColumnsList;
-if (sortColumns != null) {
-  sortColumnsList = Arrays.asList(sortColumns);
+if (!isTransactionalTable) {
--- End diff --

always directly set value to tableSchemaBuilder


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183239131
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -75,6 +78,11 @@ public TableSchemaBuilder tableName(String tableName) {
 return this;
   }
 
+  public TableSchemaBuilder resetTransactionalTable(boolean 
isTransactionalTable) {
--- End diff --

change function name to setTransactionTable , there nothing resetting 
happening in this function


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183227722
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -196,11 +235,26 @@ private CarbonTable buildCarbonTable() {
   tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize);
 }
 
-List sortColumnsList;
-if (sortColumns != null) {
-  sortColumnsList = Arrays.asList(sortColumns);
+if (isUnManagedTable) {
+  tableSchemaBuilder = 
tableSchemaBuilder.isUnmanagedTable(isUnManagedTable);
+}
+
+List sortColumnsList = new ArrayList<>();
--- End diff --

ok. done


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183227711
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) 
{
 return this;
   }
 
+  /**
+   * To support the load options for sdk writer
+   * @param options key,value pair of load options.
+   *supported keys values are
+   *a. bad_records_logger_enable -- true, false
+   *b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT
+   *c. bad_record_path -- path
+   *d. dateformat -- same as JAVA SimpleDateFormat
+   *e. timestampformat -- same as JAVA SimpleDateFormat
+   * @return updated CarbonWriterBuilder
+   */
+  public CarbonWriterBuilder withLoadOptions(Map options) {
--- End diff --

ok. done


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

Github user ajantha-bhat commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183227700
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -119,7 +126,18 @@ public TableSchemaBuilder addColumn(StructField field, 
boolean isSortColumn) {
 }
 newColumn.setSchemaOrdinal(ordinal++);
 newColumn.setColumnar(true);
-newColumn.setColumnUniqueId(UUID.randomUUID().toString());
+
+// For unmanagedTable, multiple sdk writer output with same column 
name can be placed in
--- End diff --

ok. done


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183225480
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) 
{
 return this;
   }
 
+  /**
+   * To support the load options for sdk writer
+   * @param options key,value pair of load options.
+   *supported keys values are
+   *a. bad_records_logger_enable -- true, false
--- End diff --

 please follow standard documentation format 


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183225467
  
--- Diff: 
store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java ---
@@ -48,11 +48,11 @@ public void testWriteAndReadFiles() throws IOException, 
InterruptedException {
 
 int i = 0;
 while (reader.hasNext()) {
-  Object[] row = (Object[])reader.readNextRow();
-  Assert.assertEquals("robot" + (i % 10), row[0]);
-  Assert.assertEquals(i, row[1]);
+  Object[] row = (Object[]) reader.readNextRow();
+  // TODO: Default sort column is applied for dimensions. So, need  to 
validate accordingly
--- End diff --

TODO: to be corrected


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183225463
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) 
{
 return this;
   }
 
+  /**
+   * To support the load options for sdk writer
+   * @param options key,value pair of load options.
+   *supported keys values are
+   *a. bad_records_logger_enable -- true, false
+   *b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT
+   *c. bad_record_path -- path
+   *d. dateformat -- same as JAVA SimpleDateFormat
+   *e. timestampformat -- same as JAVA SimpleDateFormat
+   * @return updated CarbonWriterBuilder
+   */
+  public CarbonWriterBuilder withLoadOptions(Map options) {
--- End diff --

complex type level delimiters should be supported, quote char, escape 
character needs to be supported, as required for complex types parsing.


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183225457
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java
 ---
@@ -119,7 +126,18 @@ public TableSchemaBuilder addColumn(StructField field, 
boolean isSortColumn) {
 }
 newColumn.setSchemaOrdinal(ordinal++);
 newColumn.setColumnar(true);
-newColumn.setColumnUniqueId(UUID.randomUUID().toString());
+
+// For unmanagedTable, multiple sdk writer output with same column 
name can be placed in
--- End diff --

Change un managed to transactional table in all PR specific code added.


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183225429
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java 
---
@@ -196,11 +235,26 @@ private CarbonTable buildCarbonTable() {
   tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize);
 }
 
-List sortColumnsList;
-if (sortColumns != null) {
-  sortColumnsList = Arrays.asList(sortColumns);
+if (isUnManagedTable) {
+  tableSchemaBuilder = 
tableSchemaBuilder.isUnmanagedTable(isUnManagedTable);
+}
+
+List sortColumnsList = new ArrayList<>();
--- End diff --

Sort options behaviour should be consistent as create table 
If empty sort columns list, then it should be no sort.
If sort column list not passed, default behaviour should be same as create 
table.


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[gvramana]

Github user gvramana commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2190#discussion_r183225423
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/executor/util/RestructureUtil.java
 ---
@@ -355,12 +338,11 @@ public static Object 
getMeasureDefaultValueByType(ColumnSchema columnSchema,
* @param blockExecutionInfo
* @param queryMeasuresmeasures present in query
* @param currentBlockMeasures current block measures
-   * @param isUnManagedTable
* @return measures present in the block
*/
   public static List 
createMeasureInfoAndGetCurrentBlockQueryMeasures(
   BlockExecutionInfo blockExecutionInfo, List 
queryMeasures,
-  List currentBlockMeasures, boolean isUnManagedTable) {
+  List currentBlockMeasures) {
--- End diff --

For non transactional table columnid check should not be persent during 
read. As files can be created some where else and can be copied.


---

[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...

[ajantha-bhat]

GitHub user ajantha-bhat opened a pull request:

https://github.com/apache/carbondata/pull/2190

[CARBONDATA-2359] Support applicable load options and table properties for 
Non-Transactional table

[CARBONDATA-2359] Support applicable load options and table properties for 
a Non-Transactional table
And blocked clean files for a Non-Transactional table.
your contribution quickly and easily:

 - [ ] Any interfaces changed? No. Added new interfaces. Didn't modified 
any.
 
 - [ ] Any backward compatibility impacted? No
 
 - [ ] Document update required? yes, will be handled in separate PR

 - [ ] Testing done.
   please refer TestUnmanagedCarbonTable.scala
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. Done.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajantha-bhat/carbondata unmanaged_table

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2190


commit 5f9aabaa7ed6498357c806617da2f00fceebbaf3
Author: ajantha-bhat 
Date:   2018-04-19T13:11:20Z

[CARBONDATA-2359] Support applicable load options and table properties for 
Non-Transactional table




---