[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2190 ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183274384 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) { return this; } + /** + * To support the load options for sdk writer + * @param options key,value pair of load options. + *supported keys values are + *a. bad_records_logger_enable -- true, false --- End diff -- ok ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183242493 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java --- @@ -75,6 +78,11 @@ public TableSchemaBuilder tableName(String tableName) { return this; } + public TableSchemaBuilder resetTransactionalTable(boolean isTransactionalTable) { --- End diff -- ok ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183242496 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -210,11 +260,26 @@ private CarbonTable buildCarbonTable() { tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize); } -List sortColumnsList; -if (sortColumns != null) { - sortColumnsList = Arrays.asList(sortColumns); +if (!isTransactionalTable) { --- End diff -- ok ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183239308 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -210,11 +260,26 @@ private CarbonTable buildCarbonTable() { tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize); } -List sortColumnsList; -if (sortColumns != null) { - sortColumnsList = Arrays.asList(sortColumns); +if (!isTransactionalTable) { --- End diff -- always directly set value to tableSchemaBuilder ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183239131 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java --- @@ -75,6 +78,11 @@ public TableSchemaBuilder tableName(String tableName) { return this; } + public TableSchemaBuilder resetTransactionalTable(boolean isTransactionalTable) { --- End diff -- change function name to setTransactionTable , there nothing resetting happening in this function ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183227722 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -196,11 +235,26 @@ private CarbonTable buildCarbonTable() { tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize); } -List sortColumnsList; -if (sortColumns != null) { - sortColumnsList = Arrays.asList(sortColumns); +if (isUnManagedTable) { + tableSchemaBuilder = tableSchemaBuilder.isUnmanagedTable(isUnManagedTable); +} + +List sortColumnsList = new ArrayList<>(); --- End diff -- ok. done ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183227711 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) { return this; } + /** + * To support the load options for sdk writer + * @param options key,value pair of load options. + *supported keys values are + *a. bad_records_logger_enable -- true, false + *b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT + *c. bad_record_path -- path + *d. dateformat -- same as JAVA SimpleDateFormat + *e. timestampformat -- same as JAVA SimpleDateFormat + * @return updated CarbonWriterBuilder + */ + public CarbonWriterBuilder withLoadOptions(Map options) { --- End diff -- ok. done ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183227700 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java --- @@ -119,7 +126,18 @@ public TableSchemaBuilder addColumn(StructField field, boolean isSortColumn) { } newColumn.setSchemaOrdinal(ordinal++); newColumn.setColumnar(true); -newColumn.setColumnUniqueId(UUID.randomUUID().toString()); + +// For unmanagedTable, multiple sdk writer output with same column name can be placed in --- End diff -- ok. done ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183225480 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) { return this; } + /** + * To support the load options for sdk writer + * @param options key,value pair of load options. + *supported keys values are + *a. bad_records_logger_enable -- true, false --- End diff -- please follow standard documentation format ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183225467 --- Diff: store/sdk/src/test/java/org/apache/carbondata/sdk/file/CarbonReaderTest.java --- @@ -48,11 +48,11 @@ public void testWriteAndReadFiles() throws IOException, InterruptedException { int i = 0; while (reader.hasNext()) { - Object[] row = (Object[])reader.readNextRow(); - Assert.assertEquals("robot" + (i % 10), row[0]); - Assert.assertEquals(i, row[1]); + Object[] row = (Object[]) reader.readNextRow(); + // TODO: Default sort column is applied for dimensions. So, need to validate accordingly --- End diff -- TODO: to be corrected ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183225463 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -123,6 +125,43 @@ public CarbonWriterBuilder uniqueIdentifier(long UUID) { return this; } + /** + * To support the load options for sdk writer + * @param options key,value pair of load options. + *supported keys values are + *a. bad_records_logger_enable -- true, false + *b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT + *c. bad_record_path -- path + *d. dateformat -- same as JAVA SimpleDateFormat + *e. timestampformat -- same as JAVA SimpleDateFormat + * @return updated CarbonWriterBuilder + */ + public CarbonWriterBuilder withLoadOptions(Map options) { --- End diff -- complex type level delimiters should be supported, quote char, escape character needs to be supported, as required for complex types parsing. ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183225457 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java --- @@ -119,7 +126,18 @@ public TableSchemaBuilder addColumn(StructField field, boolean isSortColumn) { } newColumn.setSchemaOrdinal(ordinal++); newColumn.setColumnar(true); -newColumn.setColumnUniqueId(UUID.randomUUID().toString()); + +// For unmanagedTable, multiple sdk writer output with same column name can be placed in --- End diff -- Change un managed to transactional table in all PR specific code added. ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183225429 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -196,11 +235,26 @@ private CarbonTable buildCarbonTable() { tableSchemaBuilder = tableSchemaBuilder.blockletSize(blockletSize); } -List sortColumnsList; -if (sortColumns != null) { - sortColumnsList = Arrays.asList(sortColumns); +if (isUnManagedTable) { + tableSchemaBuilder = tableSchemaBuilder.isUnmanagedTable(isUnManagedTable); +} + +List sortColumnsList = new ArrayList<>(); --- End diff -- Sort options behaviour should be consistent as create table If empty sort columns list, then it should be no sort. If sort column list not passed, default behaviour should be same as create table. ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2190#discussion_r183225423 --- Diff: core/src/main/java/org/apache/carbondata/core/scan/executor/util/RestructureUtil.java --- @@ -355,12 +338,11 @@ public static Object getMeasureDefaultValueByType(ColumnSchema columnSchema, * @param blockExecutionInfo * @param queryMeasuresmeasures present in query * @param currentBlockMeasures current block measures - * @param isUnManagedTable * @return measures present in the block */ public static List createMeasureInfoAndGetCurrentBlockQueryMeasures( BlockExecutionInfo blockExecutionInfo, List queryMeasures, - List currentBlockMeasures, boolean isUnManagedTable) { + List currentBlockMeasures) { --- End diff -- For non transactional table columnid check should not be persent during read. As files can be created some where else and can be copied. ---
[GitHub] carbondata pull request #2190: [CARBONDATA-2359] Support applicable load opt...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2190 [CARBONDATA-2359] Support applicable load options and table properties for Non-Transactional table [CARBONDATA-2359] Support applicable load options and table properties for a Non-Transactional table And blocked clean files for a Non-Transactional table. your contribution quickly and easily: - [ ] Any interfaces changed? No. Added new interfaces. Didn't modified any. - [ ] Any backward compatibility impacted? No - [ ] Document update required? yes, will be handled in separate PR - [ ] Testing done. please refer TestUnmanagedCarbonTable.scala - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. Done. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata unmanaged_table Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2190.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2190 commit 5f9aabaa7ed6498357c806617da2f00fceebbaf3 Author: ajantha-bhat Date: 2018-04-19T13:11:20Z [CARBONDATA-2359] Support applicable load options and table properties for Non-Transactional table ---