[GitHub] drill pull request #793: DRILL-4678: Tune metadata by generating a dispatche...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/793#discussion_r107647154 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/cost/DrillRelMdRowCount.java --- @@ -14,35 +14,71 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - **/ + */ package org.apache.drill.exec.planner.cost; +import org.apache.calcite.rel.SingleRel; import org.apache.calcite.rel.core.Aggregate; import org.apache.calcite.rel.core.Filter; +import org.apache.calcite.rel.core.Join; +import org.apache.calcite.rel.core.Project; +import org.apache.calcite.rel.core.Sort; +import org.apache.calcite.rel.core.Union; import org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider; import org.apache.calcite.rel.metadata.RelMdRowCount; import org.apache.calcite.rel.metadata.RelMetadataProvider; +import org.apache.calcite.rel.metadata.RelMetadataQuery; import org.apache.calcite.util.BuiltInMethod; import org.apache.calcite.util.ImmutableBitSet; +import org.apache.drill.exec.planner.common.DrillLimitRelBase; public class DrillRelMdRowCount extends RelMdRowCount{ private static final DrillRelMdRowCount INSTANCE = new DrillRelMdRowCount(); public static final RelMetadataProvider SOURCE = ReflectiveRelMetadataProvider.reflectiveSource(BuiltInMethod.ROW_COUNT.method, INSTANCE); @Override - public Double getRowCount(Aggregate rel) { + public Double getRowCount(Aggregate rel, RelMetadataQuery mq) { ImmutableBitSet groupKey = ImmutableBitSet.range(rel.getGroupCount()); if (groupKey.isEmpty()) { return 1.0; } else { - return super.getRowCount(rel); + return super.getRowCount(rel, mq); } } @Override - public Double getRowCount(Filter rel) { -return rel.getRows(); + public Double getRowCount(Filter rel, RelMetadataQuery mq) { --- End diff -- In RelMdRowCount.java added overloaded methods, which returns other result than getRowCount(RelNode rel, RelMetadataQuery mq) [1] as it was before. So in order to preserve previous Drill behavior, I override those methods. [1] https://github.com/Serhii-Harnyk/incubator-calcite/blob/02848c99d75f2d0e00c219f3fa300fc8e7df26df/core/src/main/java/org/apache/calcite/rel/metadata/RelMdRowCount.java#L65 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #793: DRILL-4678: Tune metadata by generating a dispatche...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/793 DRILL-4678: Tune metadata by generating a dispatcher at runtime Changes for rebasing to Calcite 1.4.0-drill-r20 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-4678 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/793.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #793 commit 794e9537bbe7d25abe3809db5847d898c45c73e1 Author: Serhii-Harnyk Date: 2017-03-03T15:24:26Z DRILL-4678: Tune metadata by generating a dispatcher at runtime --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #748: DRILL-5263: Prevent left NLJoin with non scalar subqueries
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/748 Merged into apace master without message _close apache/drill#748_. Closing PR manually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...
Github user Serhii-Harnyk closed the pull request at: https://github.com/apache/drill/pull/748 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r101759433 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -492,7 +538,19 @@ private void writeDataAllText(MapWriter map, FieldSelection selection, } } map.end(); + } + /** + * Puts copy of field path list to fieldPathWriter map. + * @param fieldName + */ + private void putFieldPath(String fieldName, MapWriter map) { --- End diff -- I attached results of testing on the last fix to the Jira. https://drive.google.com/open?id=1l3Dg0DCV3p-OhwA0v6qdl2wGcezXDl5qaUqv56EEBn8 It should be mentioned that a lot of time goes to the creating fields and them writers. So times of execution also compared with time of querying files, in which nulls replaced by varchar. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/748#discussion_r101308389 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/NestedLoopJoinPrule.java --- @@ -49,7 +49,7 @@ protected boolean checkPreconditions(DrillJoinRel join, RelNode left, RelNode ri PlannerSettings settings) { JoinRelType type = join.getJoinType(); -if (! (type == JoinRelType.INNER || type == JoinRelType.LEFT)) { +if (!(type == JoinRelType.INNER || (type == JoinRelType.LEFT && JoinUtils.hasScalarSubqueryInput(left, right { --- End diff -- Yes, you a right, we should return error in this case too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/748#discussion_r101086894 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/NestedLoopJoinPrule.java --- @@ -49,7 +49,7 @@ protected boolean checkPreconditions(DrillJoinRel join, RelNode left, RelNode ri PlannerSettings settings) { JoinRelType type = join.getJoinType(); -if (! (type == JoinRelType.INNER || type == JoinRelType.LEFT)) { +if (!(type == JoinRelType.INNER || (type == JoinRelType.LEFT && JoinUtils.hasScalarSubqueryInput(left, right { --- End diff -- In the case when table joining with the result of the query with aggregate function, Drill creates physical plan with left nested loop join. It is due to this code line: https://github.com/mapr/incubator-calcite/blob/DrillCalcite1.4.0/core/src/main/java/org/apache/calcite/sql2rel/SqlToRelConverter.java#L1078 So to allow execution of such queries without changes in calcite - allow left nested loop join for scalar subqueries. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 @amansinha100, could you please review new changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r101024133 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -492,10 +537,18 @@ private void writeDataAllText(MapWriter map, FieldSelection selection, } } map.end(); - } /** + * Puts copy of field path list to fieldPathWriter map. + * @param fieldName + */ + private void putFieldPath(String fieldName, MapWriter map) { +List fieldPath = Lists.newArrayList(path); --- End diff -- Thanks, fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #748: DRILL-5263: Prevent left NLJoin with non scalar sub...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/748 DRILL-5263: Prevent left NLJoin with non scalar subqueries You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-5263 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/748.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #748 commit fff774ed62bf85f36a0167b1e8c49c1339312131 Author: Serhii-Harnyk Date: 2017-02-13T14:30:27Z DRILL-5263: Prevent left NLJoin with non scalar subqueries --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-5263) Prevent left NLJoin with non scalar subqueries
Serhii Harnyk created DRILL-5263: Summary: Prevent left NLJoin with non scalar subqueries Key: DRILL-5263 URL: https://issues.apache.org/jira/browse/DRILL-5263 Project: Apache Drill Issue Type: Bug Reporter: Serhii Harnyk Assignee: Serhii Harnyk Nested loop join operator in Drill supports only inner join and returns incorrect result for queries with left join and non scalar sub-queries. Drill should throw error in this case. Example: {code:sql} alter session set planner.enable_nljoin_for_scalar_only=false; select t2.dt, t1.fyq, t2.who, t2.event from t2 left join t1 on t2.dt between t1.dts and t1.dte order by t2.dt; {code} Result: {noformat} +-+--+--++ | dt | fyq| who| event| +-+--+--++ | 2016-12-26 | 2016-Q2 | aperson | had chrsitmas | | 2017-01-06 | 2016-Q3 | aperson | did somthing | | 2017-01-12 | 2016-Q3 | aperson | did somthing else | +-+--+--++ {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 Squashed all changes into single commit, rebased on master and resolved conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/581 @paul-rogers could you please review new changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99315319 --- Diff: logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java --- @@ -0,0 +1,216 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import com.google.common.collect.Sets; +import org.apache.commons.lang3.StringUtils; +import org.apache.drill.common.map.CaseInsensitiveMap; + +import java.util.Comparator; +import java.util.Set; + +public class JodaDateValidator { + + private static final Set postgresValuesForDeleting = Sets.newTreeSet(new LengthDescComparator()); + private static final CaseInsensitiveMap postgresToJodaMap = CaseInsensitiveMap.newTreeMap(new LengthDescComparator()); + + // tokens for deleting + public static final String SUFFIX_SP = "sp"; + public static final String PREFIX_FM = "fm"; + public static final String PREFIX_FX = "fx"; + public static final String PREFIX_TM = "tm"; + + // postgres patterns + public static final String POSTGRES_FULL_NAME_OF_DAY = "day"; + public static final String POSTGRES_DAY_OF_YEAR = "ddd"; + public static final String POSTGRES_DAY_OF_MONTH = "dd"; + public static final String POSTGRES_DAY_OF_WEEK = "d"; + public static final String POSTGRES_NAME_OF_MONTH = "month"; + public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon"; + public static final String POSTGRES_FULL_ERA_NAME = "ee"; + public static final String POSTGRES_NAME_OF_DAY = "dy"; + public static final String POSTGRES_TIME_ZONE_NAME = "tz"; + public static final String POSTGRES_HOUR_12_NAME = "hh"; + public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12"; + public static final String POSTGRES_HOUR_24_NAME = "hh24"; + public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi"; + public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss"; + public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms"; + public static final String POSTGRES_WEEK_OF_YEAR = "ww"; + public static final String POSTGRES_MONTH = "mm"; + public static final String POSTGRES_HALFDAY_AM = "am"; + public static final String POSTGRES_HALFDAY_PM = "pm"; + + // jodaTime patterns + public static final String JODA_FULL_NAME_OF_DAY = ""; + public static final String JODA_DAY_OF_YEAR = "D"; + public static final String JODA_DAY_OF_MONTH = "d"; + public static final String JODA_DAY_OF_WEEK = "e"; + public static final String JODA_NAME_OF_MONTH = ""; + public static final String JODA_ABR_NAME_OF_MONTH = "MMM"; + public static final String JODA_FULL_ERA_NAME = "G"; + public static final String JODA_NAME_OF_DAY = "E"; + public static final String JODA_TIME_ZONE_NAME = "TZ"; + public static final String JODA_HOUR_12_NAME = "h"; + public static final String JODA_HOUR_12_OTHER_NAME = "h"; + public static final String JODA_HOUR_24_NAME = "H"; + public static final String JODA_MINUTE_OF_HOUR_NAME = "m"; + public static final String JODA_SECOND_OF_MINUTE_NAME = "s"; + public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS"; + public static final String JODA_WEEK_OF_YEAR = "w"; + public static final String JODA_MONTH = "MM"; + public static final String JODA_HALFDAY = "aa"; + + static { +postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY); +postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR); +postgresToJod
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99313550 --- Diff: logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java --- @@ -0,0 +1,216 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import com.google.common.collect.Sets; +import org.apache.commons.lang3.StringUtils; +import org.apache.drill.common.map.CaseInsensitiveMap; + +import java.util.Comparator; +import java.util.Set; + +public class JodaDateValidator { + + private static final Set postgresValuesForDeleting = Sets.newTreeSet(new LengthDescComparator()); + private static final CaseInsensitiveMap postgresToJodaMap = CaseInsensitiveMap.newTreeMap(new LengthDescComparator()); + + // tokens for deleting + public static final String SUFFIX_SP = "sp"; + public static final String PREFIX_FM = "fm"; + public static final String PREFIX_FX = "fx"; + public static final String PREFIX_TM = "tm"; + + // postgres patterns + public static final String POSTGRES_FULL_NAME_OF_DAY = "day"; + public static final String POSTGRES_DAY_OF_YEAR = "ddd"; + public static final String POSTGRES_DAY_OF_MONTH = "dd"; + public static final String POSTGRES_DAY_OF_WEEK = "d"; + public static final String POSTGRES_NAME_OF_MONTH = "month"; + public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon"; + public static final String POSTGRES_FULL_ERA_NAME = "ee"; + public static final String POSTGRES_NAME_OF_DAY = "dy"; + public static final String POSTGRES_TIME_ZONE_NAME = "tz"; + public static final String POSTGRES_HOUR_12_NAME = "hh"; + public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12"; + public static final String POSTGRES_HOUR_24_NAME = "hh24"; + public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi"; + public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss"; + public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms"; + public static final String POSTGRES_WEEK_OF_YEAR = "ww"; + public static final String POSTGRES_MONTH = "mm"; + public static final String POSTGRES_HALFDAY_AM = "am"; + public static final String POSTGRES_HALFDAY_PM = "pm"; + + // jodaTime patterns + public static final String JODA_FULL_NAME_OF_DAY = ""; + public static final String JODA_DAY_OF_YEAR = "D"; + public static final String JODA_DAY_OF_MONTH = "d"; + public static final String JODA_DAY_OF_WEEK = "e"; + public static final String JODA_NAME_OF_MONTH = ""; + public static final String JODA_ABR_NAME_OF_MONTH = "MMM"; + public static final String JODA_FULL_ERA_NAME = "G"; + public static final String JODA_NAME_OF_DAY = "E"; + public static final String JODA_TIME_ZONE_NAME = "TZ"; + public static final String JODA_HOUR_12_NAME = "h"; + public static final String JODA_HOUR_12_OTHER_NAME = "h"; + public static final String JODA_HOUR_24_NAME = "H"; + public static final String JODA_MINUTE_OF_HOUR_NAME = "m"; + public static final String JODA_SECOND_OF_MINUTE_NAME = "s"; + public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS"; + public static final String JODA_WEEK_OF_YEAR = "w"; + public static final String JODA_MONTH = "MM"; + public static final String JODA_HALFDAY = "aa"; + + static { +postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY); +postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR); +postgresToJod
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99317199 --- Diff: logical/src/test/java/org/apache/drill/common/expression/fn/JodaDateValidatorTest.java --- @@ -0,0 +1,176 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import org.joda.time.DateTime; +import org.joda.time.DateTimeZone; +import org.joda.time.format.DateTimeFormatter; +import org.junit.Assert; +import org.junit.Test; + +import static org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat; +import static org.joda.time.DateTime.parse; +import static org.joda.time.format.DateTimeFormat.forPattern; + +public class JodaDateValidatorTest { --- End diff -- Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99315665 --- Diff: logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java --- @@ -0,0 +1,216 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import com.google.common.collect.Sets; +import org.apache.commons.lang3.StringUtils; +import org.apache.drill.common.map.CaseInsensitiveMap; + +import java.util.Comparator; +import java.util.Set; + +public class JodaDateValidator { + + private static final Set postgresValuesForDeleting = Sets.newTreeSet(new LengthDescComparator()); + private static final CaseInsensitiveMap postgresToJodaMap = CaseInsensitiveMap.newTreeMap(new LengthDescComparator()); + + // tokens for deleting + public static final String SUFFIX_SP = "sp"; + public static final String PREFIX_FM = "fm"; + public static final String PREFIX_FX = "fx"; + public static final String PREFIX_TM = "tm"; + + // postgres patterns + public static final String POSTGRES_FULL_NAME_OF_DAY = "day"; + public static final String POSTGRES_DAY_OF_YEAR = "ddd"; + public static final String POSTGRES_DAY_OF_MONTH = "dd"; + public static final String POSTGRES_DAY_OF_WEEK = "d"; + public static final String POSTGRES_NAME_OF_MONTH = "month"; + public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon"; + public static final String POSTGRES_FULL_ERA_NAME = "ee"; + public static final String POSTGRES_NAME_OF_DAY = "dy"; + public static final String POSTGRES_TIME_ZONE_NAME = "tz"; + public static final String POSTGRES_HOUR_12_NAME = "hh"; + public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12"; + public static final String POSTGRES_HOUR_24_NAME = "hh24"; + public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi"; + public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss"; + public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms"; + public static final String POSTGRES_WEEK_OF_YEAR = "ww"; + public static final String POSTGRES_MONTH = "mm"; + public static final String POSTGRES_HALFDAY_AM = "am"; + public static final String POSTGRES_HALFDAY_PM = "pm"; + + // jodaTime patterns + public static final String JODA_FULL_NAME_OF_DAY = ""; + public static final String JODA_DAY_OF_YEAR = "D"; + public static final String JODA_DAY_OF_MONTH = "d"; + public static final String JODA_DAY_OF_WEEK = "e"; + public static final String JODA_NAME_OF_MONTH = ""; + public static final String JODA_ABR_NAME_OF_MONTH = "MMM"; + public static final String JODA_FULL_ERA_NAME = "G"; + public static final String JODA_NAME_OF_DAY = "E"; + public static final String JODA_TIME_ZONE_NAME = "TZ"; + public static final String JODA_HOUR_12_NAME = "h"; + public static final String JODA_HOUR_12_OTHER_NAME = "h"; + public static final String JODA_HOUR_24_NAME = "H"; + public static final String JODA_MINUTE_OF_HOUR_NAME = "m"; + public static final String JODA_SECOND_OF_MINUTE_NAME = "s"; + public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS"; + public static final String JODA_WEEK_OF_YEAR = "w"; + public static final String JODA_MONTH = "MM"; + public static final String JODA_HALFDAY = "aa"; + + static { +postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY); +postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR); +postgresToJod
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99316546 --- Diff: logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java --- @@ -0,0 +1,216 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import com.google.common.collect.Sets; +import org.apache.commons.lang3.StringUtils; +import org.apache.drill.common.map.CaseInsensitiveMap; + +import java.util.Comparator; +import java.util.Set; + +public class JodaDateValidator { + + private static final Set postgresValuesForDeleting = Sets.newTreeSet(new LengthDescComparator()); + private static final CaseInsensitiveMap postgresToJodaMap = CaseInsensitiveMap.newTreeMap(new LengthDescComparator()); + + // tokens for deleting + public static final String SUFFIX_SP = "sp"; + public static final String PREFIX_FM = "fm"; + public static final String PREFIX_FX = "fx"; + public static final String PREFIX_TM = "tm"; + + // postgres patterns + public static final String POSTGRES_FULL_NAME_OF_DAY = "day"; + public static final String POSTGRES_DAY_OF_YEAR = "ddd"; + public static final String POSTGRES_DAY_OF_MONTH = "dd"; + public static final String POSTGRES_DAY_OF_WEEK = "d"; + public static final String POSTGRES_NAME_OF_MONTH = "month"; + public static final String POSTGRES_ABR_NAME_OF_MONTH = "mon"; + public static final String POSTGRES_FULL_ERA_NAME = "ee"; + public static final String POSTGRES_NAME_OF_DAY = "dy"; + public static final String POSTGRES_TIME_ZONE_NAME = "tz"; + public static final String POSTGRES_HOUR_12_NAME = "hh"; + public static final String POSTGRES_HOUR_12_OTHER_NAME = "hh12"; + public static final String POSTGRES_HOUR_24_NAME = "hh24"; + public static final String POSTGRES_MINUTE_OF_HOUR_NAME = "mi"; + public static final String POSTGRES_SECOND_OF_MINUTE_NAME = "ss"; + public static final String POSTGRES_MILLISECOND_OF_MINUTE_NAME = "ms"; + public static final String POSTGRES_WEEK_OF_YEAR = "ww"; + public static final String POSTGRES_MONTH = "mm"; + public static final String POSTGRES_HALFDAY_AM = "am"; + public static final String POSTGRES_HALFDAY_PM = "pm"; + + // jodaTime patterns + public static final String JODA_FULL_NAME_OF_DAY = ""; + public static final String JODA_DAY_OF_YEAR = "D"; + public static final String JODA_DAY_OF_MONTH = "d"; + public static final String JODA_DAY_OF_WEEK = "e"; + public static final String JODA_NAME_OF_MONTH = ""; + public static final String JODA_ABR_NAME_OF_MONTH = "MMM"; + public static final String JODA_FULL_ERA_NAME = "G"; + public static final String JODA_NAME_OF_DAY = "E"; + public static final String JODA_TIME_ZONE_NAME = "TZ"; + public static final String JODA_HOUR_12_NAME = "h"; + public static final String JODA_HOUR_12_OTHER_NAME = "h"; + public static final String JODA_HOUR_24_NAME = "H"; + public static final String JODA_MINUTE_OF_HOUR_NAME = "m"; + public static final String JODA_SECOND_OF_MINUTE_NAME = "s"; + public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "SSS"; + public static final String JODA_WEEK_OF_YEAR = "w"; + public static final String JODA_MONTH = "MM"; + public static final String JODA_HALFDAY = "aa"; + + static { +postgresToJodaMap.put(POSTGRES_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY); +postgresToJodaMap.put(POSTGRES_DAY_OF_YEAR, JODA_DAY_OF_YEAR); +postgresToJod
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99312234 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/fn/impl/testing/TestDateConversions.java --- @@ -0,0 +1,200 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.exec.fn.impl.testing; + +import org.apache.drill.BaseTestQuery; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; + +import java.io.BufferedWriter; +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; + +public class TestDateConversions extends BaseTestQuery { --- End diff -- Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99312091 --- Diff: exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java --- @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import org.apache.drill.exec.expr.annotations.Workspace; + +<@pp.dropOutputFile/> + +<#list dateIntervalFunc.dates as type> + +<@pp.changeOutputFile name = "/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/> + +<#include "/@includes/license.ftl"/> + +package org.apache.drill.exec.expr.fn.impl; + +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Workspace; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.*; + +/* + * This class is generated using freemarker and the ${.template_name} template. + */ + +@FunctionTemplate(name = "sql_to_${type?lower_case}", + scope = FunctionTemplate.FunctionScope.SIMPLE, + nulls = NullHandling.NULL_IF_NULL) +public class SqlTo${type} implements DrillSimpleFunc { + +@Param VarCharHolder left; +@Param VarCharHolder right; +@Workspace org.joda.time.format.DateTimeFormatter format; +@Output ${type}Holder out; + +public void setup() { +// Get the desired output format +byte[] buf = new byte[right.end - right.start]; +right.buffer.getBytes(right.start, buf, 0, right.end - right.start); +String formatString = new String(buf, com.google.common.base.Charsets.UTF_8); +String pattern = org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat(formatString); +format = org.joda.time.format.DateTimeFormat.forPattern(pattern); +} + +public void eval() { +// Get the input +byte[] buf1 = new byte[left.end - left.start]; +left.buffer.getBytes(left.start, buf1, 0, left.end - left.start); +String input = new String(buf1, com.google.common.base.Charsets.UTF_8); + +<#if type == "Date"> +out.value = (org.joda.time.DateMidnight.parse(input, format).withZoneRetainFields(org.joda.time.DateTimeZone.UTC)).getMillis(); +<#elseif type == "TimeStamp"> +out.value = org.joda.time.DateTime.parse(input, format).withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis(); +<#elseif type == "Time"> +out.value = (int) ((format.parseDateTime(input)).withZoneRetainFields(org.joda.time.DateTimeZone.UTC).getMillis()); + --- End diff -- Yes, we should. Fixed, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99311783 --- Diff: exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java --- @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import org.apache.drill.exec.expr.annotations.Workspace; + +<@pp.dropOutputFile/> + +<#list dateIntervalFunc.dates as type> + +<@pp.changeOutputFile name = "/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/> + +<#include "/@includes/license.ftl"/> + +package org.apache.drill.exec.expr.fn.impl; + +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Workspace; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.*; + +/* + * This class is generated using freemarker and the ${.template_name} template. + */ + +@FunctionTemplate(name = "sql_to_${type?lower_case}", + scope = FunctionTemplate.FunctionScope.SIMPLE, + nulls = NullHandling.NULL_IF_NULL) +public class SqlTo${type} implements DrillSimpleFunc { + +@Param VarCharHolder left; +@Param VarCharHolder right; +@Workspace org.joda.time.format.DateTimeFormatter format; +@Output ${type}Holder out; + +public void setup() { +// Get the desired output format +byte[] buf = new byte[right.end - right.start]; +right.buffer.getBytes(right.start, buf, 0, right.end - right.start); +String formatString = new String(buf, com.google.common.base.Charsets.UTF_8); +String pattern = org.apache.drill.common.expression.fn.JodaDateValidator.toJodaFormat(formatString); --- End diff -- Before the using Java 8 classes, we need change some patterns in JodaDateValidator.class, because there are some differences, but in general there shouldn't be user-visible change in behavior or semantics. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r99311060 --- Diff: exec/java-exec/src/main/codegen/templates/DateIntervalFunctionTemplates/SqlToDateTypeFunctions.java --- @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import org.apache.drill.exec.expr.annotations.Workspace; + +<@pp.dropOutputFile/> + +<#list dateIntervalFunc.dates as type> + +<@pp.changeOutputFile name = "/org/apache/drill/exec/expr/fn/impl/SqlTo${type}.java"/> + +<#include "/@includes/license.ftl"/> + +package org.apache.drill.exec.expr.fn.impl; + +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Workspace; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.*; + +/* + * This class is generated using freemarker and the ${.template_name} template. + */ + +@FunctionTemplate(name = "sql_to_${type?lower_case}", + scope = FunctionTemplate.FunctionScope.SIMPLE, + nulls = NullHandling.NULL_IF_NULL) +public class SqlTo${type} implements DrillSimpleFunc { + +@Param VarCharHolder left; +@Param VarCharHolder right; +@Workspace org.joda.time.format.DateTimeFormatter format; +@Output ${type}Holder out; + +public void setup() { +// Get the desired output format +byte[] buf = new byte[right.end - right.start]; +right.buffer.getBytes(right.start, buf, 0, right.end - right.start); +String formatString = new String(buf, com.google.common.base.Charsets.UTF_8); --- End diff -- I found the method that does the same. Fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #735: DRILL-5237: Same query produces different/wrong res...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/735 DRILL-5237: Same query produces different/wrong results Fixed loss of nested fields from the schema when the first FlattenRecordBatch is empty You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-5237 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #735 commit 800953b7ec924ea31ffc21327950c84528f3c492 Author: Serhii-Harnyk Date: 2017-01-27T15:36:10Z DRILL-5237: Same query produces different/wrong results --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-5237) Same query produces different/wrong results
Serhii Harnyk created DRILL-5237: Summary: Same query produces different/wrong results Key: DRILL-5237 URL: https://issues.apache.org/jira/browse/DRILL-5237 Project: Apache Drill Issue Type: Bug Reporter: Serhii Harnyk Assignee: Serhii Harnyk Query {code:sql} select count(*) as col from (select t1.a.a1 from (select t.*, flatten(t.b) as b from dfs.`/tmp/test_data` t where t.c is not null) t1 where t1.a .a1 like 'a1'); {code} returns different results for different values for option planner.width.max_per_node With options {code:sql} alter session set `planner.slice_target` = 1; alter session set `planner.width.max_per_node` = 1; {code} query returns correct result: {noformat} +--+ | col | +--+ | 3| +--+ {noformat} but with options {code:sql} alter session set `planner.slice_target` = 1; alter session set `planner.width.max_per_node` = 3; {code} the same query returns wrong result {noformat} +--+ | col | +--+ | 2| +--+ {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] drill pull request #580: DRILL-4824: JSON with complex nested data produces ...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/580#discussion_r98232198 --- Diff: exec/vector/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java --- @@ -317,6 +317,12 @@ public Object getObject(int index) { if (v != null && index < v.getAccessor().getValueCount()) { Object value = v.getAccessor().getObject(index); if (value != null) { +if ((v.getAccessor().getObject(index) instanceof Map +&& ((Map) v.getAccessor().getObject(index)).size() == 0) +|| (v.getAccessor().getObject(index) instanceof List +&& ((List) v.getAccessor().getObject(index)).size() == 0)) { + continue; +} --- End diff -- @paul-rogers, map fields have data mode required and they are the part of the schema, that's why there are no difference between missing field in some record, and the field that exists but empty. This fix for your example will return result `{"a":{"b":10},"c":[1,2,3]}` `{"c":[4]}` `{"b":[5]}` `{"a":{"b":20}}` `{"a":{"b":20}}` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #673: DRILL-4764: Parquet file with INT_16, etc. logical types n...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/673 Squashed all changes into single commit and rebased on master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #713: DRILL-3562: Query fails when using flatten on JSON data wh...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/713 @amansinha100, could you please review new changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #713: DRILL-3562: Query fails when using flatten on JSON ...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/713#discussion_r97587774 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -59,6 +59,12 @@ private final boolean readNumbersAsDouble; /** + * Collection for tracking empty array writers during reading + * and storing them for initializing empty arrays + */ + private final Set emptyArrayWritersSet = Sets.newHashSet(); --- End diff -- But it should be mentioned that this list may contain many duplicates, as every ListWriter from every empty array field will be added into it. To avoid secondary call field.integer() on the same ListWriter object, which was added many times into the list, in the method JsonReader.ensureAtLeastOneField() we check valueCapacity of every ListWriter. Also LinkedHashSet may be used there, but in the case of many empty array fields it may cause additional extra cost. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #713: DRILL-3562: Query fails when using flatten on JSON ...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/713#discussion_r97576603 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -59,6 +59,12 @@ private final boolean readNumbersAsDouble; /** + * Collection for tracking empty array writers during reading + * and storing them for initializing empty arrays + */ + private final Set emptyArrayWritersSet = Sets.newHashSet(); --- End diff -- @amansinha100 Yes, you are right, in this place should be used List. Fixed it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #713: DRILL-3562: Query fails when using flatten on JSON ...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/713 DRILL-3562: Query fails when using flatten on JSON data where some do… …cuments have an empty array 1. Added set for ListWriters tracking to keep empty arrays for further initializing in ensureAtLeastOneField method. 2. Added check to avoid schema generating with field type "Late" and mode "Optional", replaced it to "Int" type in FlattenRecordBatch class. 3. Added unit tests to cover cases querying Json with empty arrays with flatten. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-3562 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/713.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #713 commit 815de0cc6f16b247b3a655007241a074e38394c7 Author: Serhii-Harnyk Date: 2016-12-20T16:55:41Z DRILL-3562: Query fails when using flatten on JSON data where some documents have an empty array --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #711: DRILL-5164: Equi-join query results in CompileExcep...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/711 DRILL-5164: Equi-join query results in CompileException 1. Added method call rotateBlock() to avoid adding expressions into single Jblock. 2. Added unit tests to cover three types of joins: hash join, merge join, nested loop join. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-5164 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #711 commit c448c5d4b4d8040dcf848b2478a3e833d9839d65 Author: Serhii-Harnyk Date: 2016-12-27T16:20:37Z DRILL-5164: Equi-join query results in CompileException --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (DRILL-1808) Large compilation unit tests fails due to high memory allocation
[ https://issues.apache.org/jira/browse/DRILL-1808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Serhii Harnyk resolved DRILL-1808. -- Resolution: Duplicate Fix Version/s: (was: Future) 1.10.0 Fixed in [810198b|https://github.com/apache/drill/commit/810198b18bfbe38712256c58b102bca079d934c1] > Large compilation unit tests fails due to high memory allocation > > > Key: DRILL-1808 > URL: https://issues.apache.org/jira/browse/DRILL-1808 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Codegen >Reporter: Steven Phillips >Priority: Minor > Fix For: 1.10.0 > > > The buildSchema method of external sort calls allocateNew() on each vector in > schema batch, which normally isn't much memory, but when there are thousands > of columns, this ends up being significant. We should avoid allocating any > memory for the schema batch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 @chunhui-shi, I have made some changes, could you take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #654: DRILL-5032: Drill query on hive parquet table failed with ...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/654 @jinfengni, I have made some changes. Could you please review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/654#discussion_r92801855 --- Diff: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java --- @@ -0,0 +1,483 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.store.hive; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Order; +import org.apache.hadoop.hive.metastore.api.Partition; +import org.apache.hadoop.hive.metastore.api.SerDeInfo; +import org.apache.hadoop.hive.metastore.api.StorageDescriptor; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import com.google.common.collect.Lists; + +@JsonTypeName("table") +public class HiveTableWrapper { + + @JsonIgnore + private HiveTable table; + + @JsonProperty + public String tableName; + @JsonProperty + public String dbName; + @JsonProperty + public String owner; + @JsonProperty + public int createTime; + @JsonProperty + public int lastAccessTime; + @JsonProperty + public int retention; + @JsonProperty + public StorageDescriptorWrapper sd; + @JsonProperty + public List partitionKeys; + @JsonProperty + public Map parameters; + @JsonProperty + public String viewOriginalText; + @JsonProperty + public String viewExpandedText; + @JsonProperty + public String tableType; + @JsonProperty + public ColumnsCacheWrapper columnsCache; + + @JsonIgnore + public final Map partitionNameTypeMap = new HashMap<>(); + + @JsonCreator + public HiveTableWrapper(@JsonProperty("tableName") String tableName, @JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner, + @JsonProperty("createTime") int createTime, @JsonProperty("lastAccessTime") int lastAccessTime, + @JsonProperty("retention") int retention, @JsonProperty("sd") StorageDescriptorWrapper sd, + @JsonProperty("partitionKeys") List partitionKeys, @JsonProperty("parameters") Map parameters, + @JsonProperty("viewOriginalText") String viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText, + @JsonProperty("tableType") String tableType, @JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache + ) { +this.tableName = tableName; +this.dbName = dbName; +this.owner = owner; +this.createTime = createTime; +this.lastAccessTime = lastAccessTime; +this.retention = retention; +this.sd = sd; +this.partitionKeys = partitionKeys; +this.parameters = parameters; +this.viewOriginalText = viewOriginalText; +this.viewExpandedText = viewExpandedText; +this.tableType = tableType; +this.columnsCache = columnsCache; + +List partitionKeysUnwrapped = Lists.newArrayList(); +for (FieldSchemaWrapper w : partitionKeys) { + partitionKeysUnwrapped.add(w.getFieldSchema()); + partitionNameTypeMap.put(w.name, w.type); +} +StorageDescriptor sdUnwrapped = sd.getSd(); +this.table = new HiveTable(tableName, dbName, owner, createTime, lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped, +parameters, viewOriginalText, viewExpandedText, tableType, columnsCache.getColumnListsCache()); + } + + public Hiv
[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/654#discussion_r92801426 --- Diff: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java --- @@ -0,0 +1,483 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.store.hive; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Order; +import org.apache.hadoop.hive.metastore.api.Partition; +import org.apache.hadoop.hive.metastore.api.SerDeInfo; +import org.apache.hadoop.hive.metastore.api.StorageDescriptor; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import com.google.common.collect.Lists; + +@JsonTypeName("table") +public class HiveTableWrapper { + + @JsonIgnore + private HiveTable table; + + @JsonProperty + public String tableName; + @JsonProperty + public String dbName; + @JsonProperty + public String owner; + @JsonProperty + public int createTime; + @JsonProperty + public int lastAccessTime; + @JsonProperty + public int retention; + @JsonProperty + public StorageDescriptorWrapper sd; + @JsonProperty + public List partitionKeys; + @JsonProperty + public Map parameters; + @JsonProperty + public String viewOriginalText; + @JsonProperty + public String viewExpandedText; + @JsonProperty + public String tableType; + @JsonProperty + public ColumnsCacheWrapper columnsCache; + + @JsonIgnore + public final Map partitionNameTypeMap = new HashMap<>(); + + @JsonCreator + public HiveTableWrapper(@JsonProperty("tableName") String tableName, @JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner, + @JsonProperty("createTime") int createTime, @JsonProperty("lastAccessTime") int lastAccessTime, + @JsonProperty("retention") int retention, @JsonProperty("sd") StorageDescriptorWrapper sd, + @JsonProperty("partitionKeys") List partitionKeys, @JsonProperty("parameters") Map parameters, + @JsonProperty("viewOriginalText") String viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText, + @JsonProperty("tableType") String tableType, @JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache + ) { +this.tableName = tableName; +this.dbName = dbName; +this.owner = owner; +this.createTime = createTime; +this.lastAccessTime = lastAccessTime; +this.retention = retention; +this.sd = sd; +this.partitionKeys = partitionKeys; +this.parameters = parameters; +this.viewOriginalText = viewOriginalText; +this.viewExpandedText = viewExpandedText; +this.tableType = tableType; +this.columnsCache = columnsCache; + +List partitionKeysUnwrapped = Lists.newArrayList(); +for (FieldSchemaWrapper w : partitionKeys) { + partitionKeysUnwrapped.add(w.getFieldSchema()); + partitionNameTypeMap.put(w.name, w.type); +} +StorageDescriptor sdUnwrapped = sd.getSd(); +this.table = new HiveTable(tableName, dbName, owner, createTime, lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped, +parameters, viewOriginalText, viewExpandedText, tableType, columnsCache.getColumnListsCache()); + } + + public Hiv
[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/654#discussion_r92800877 --- Diff: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTableWrapper.java --- @@ -0,0 +1,483 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.store.hive; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Order; +import org.apache.hadoop.hive.metastore.api.Partition; +import org.apache.hadoop.hive.metastore.api.SerDeInfo; +import org.apache.hadoop.hive.metastore.api.StorageDescriptor; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.fasterxml.jackson.annotation.JsonTypeName; +import com.google.common.collect.Lists; + +@JsonTypeName("table") +public class HiveTableWrapper { + + @JsonIgnore + private HiveTable table; + + @JsonProperty + public String tableName; + @JsonProperty + public String dbName; + @JsonProperty + public String owner; + @JsonProperty + public int createTime; + @JsonProperty + public int lastAccessTime; + @JsonProperty + public int retention; + @JsonProperty + public StorageDescriptorWrapper sd; + @JsonProperty + public List partitionKeys; + @JsonProperty + public Map parameters; + @JsonProperty + public String viewOriginalText; + @JsonProperty + public String viewExpandedText; + @JsonProperty + public String tableType; + @JsonProperty + public ColumnsCacheWrapper columnsCache; + + @JsonIgnore + public final Map partitionNameTypeMap = new HashMap<>(); + + @JsonCreator + public HiveTableWrapper(@JsonProperty("tableName") String tableName, @JsonProperty("dbName") String dbName, @JsonProperty("owner") String owner, + @JsonProperty("createTime") int createTime, @JsonProperty("lastAccessTime") int lastAccessTime, + @JsonProperty("retention") int retention, @JsonProperty("sd") StorageDescriptorWrapper sd, + @JsonProperty("partitionKeys") List partitionKeys, @JsonProperty("parameters") Map parameters, + @JsonProperty("viewOriginalText") String viewOriginalText, @JsonProperty("viewExpandedText") String viewExpandedText, + @JsonProperty("tableType") String tableType, @JsonProperty("columnsCache") ColumnsCacheWrapper columnsCache + ) { +this.tableName = tableName; +this.dbName = dbName; +this.owner = owner; +this.createTime = createTime; +this.lastAccessTime = lastAccessTime; +this.retention = retention; +this.sd = sd; +this.partitionKeys = partitionKeys; +this.parameters = parameters; +this.viewOriginalText = viewOriginalText; +this.viewExpandedText = viewExpandedText; +this.tableType = tableType; +this.columnsCache = columnsCache; + +List partitionKeysUnwrapped = Lists.newArrayList(); +for (FieldSchemaWrapper w : partitionKeys) { + partitionKeysUnwrapped.add(w.getFieldSchema()); + partitionNameTypeMap.put(w.name, w.type); +} +StorageDescriptor sdUnwrapped = sd.getSd(); +this.table = new HiveTable(tableName, dbName, owner, createTime, lastAccessTime, retention, sdUnwrapped, partitionKeysUnwrapped, +parameters, viewOriginalText, viewExpandedText, tableType, columnsCache.getColumnListsCache()); + } + + public Hiv
[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/581 @paul-rogers, I have implemented changes that you have proposed. Could you take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r92478836 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -510,10 +517,13 @@ private void writeDataAllText(MapWriter map, FieldSelection selection, case VALUE_NUMBER_FLOAT: case VALUE_NUMBER_INT: case VALUE_STRING: +removeNotNullColumn(fieldName); --- End diff -- To clean 'path' if it knows it is recovering from some errors, was added this line: https://github.com/Serhii-Harnyk/drill/blob/9ce6d56b46bcd540697c85cd2f280831dd50b277/exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java#L243. Call of the method removeNotNullColumn() is a try to optimize the final size of the map fieldPathWriter, by removing fields, which were added to the map and initializing at current iteration. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #686: DRILL-5117: Compile error when query a json file with 1000...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/686 @sudheeshkatkam, when running for example test testEXTERNAL_SORT(), generates class CopierGen4, in which methods doSetup() and doEval() does not splits correctly. But with this fix them both splits into the smaller methods. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #686: DRILL-5117: Compile error when query a json file wi...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/686#discussion_r92452238 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/compile/TestLargeFileCompilation.java --- @@ -154,4 +158,20 @@ public void testProject() throws Exception { testNoResult(ITERATION_COUNT, LARGE_QUERY_SELECT_LIST); } + @Test + public void testSelectAllFromFileWithManyColumns() throws Exception { +File path = new File(BaseTestQuery.getTempDir("json/input")); --- End diff -- You are right, with this changes both tests testEXTERNAL_SORT and testTOP_N_SORT work. So I enabled them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #686: DRILL-5117: Compile error when query a json file with 1000...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/686 @jinfengni, could you please review new changes? I squashed all changes into single commit and rebased into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #689: DRILL-4938: Need better error message
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/689 @zfong, I have added the unit test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #689: DRILL-4938: Need better error message
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/689 DRILL-4938: Need better error message You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-4938 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/689.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #689 commit 89b671ebc690c19f92f5325de7df1934dd4544e5 Author: Serhii-Harnyk Date: 2016-11-18T16:02:04Z DRILL-4938: Need better error message --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #673: DRILL-4764: Parquet file with INT_16, etc. logical ...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/673#discussion_r91753304 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetFixedWidthDictionaryReaders.java --- @@ -56,6 +58,31 @@ protected void readField(long recordsToReadInThisPass) { } --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 @chunhui-shi, could you please review new changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #686: DRILL-5117: Compile error when query a json file wi...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/686 DRILL-5117: Compile error when query a json file with 1000+columns You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-5117 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/686.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #686 commit 00eaf30fd662530d8bd62059b85b0ad179768fdb Author: Serhii-Harnyk Date: 2016-12-08T20:08:34Z DRILL-5117: Compile error when query a json file with 1000+columns --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-5117) Compile error when query a json file with 1000+columns
Serhii Harnyk created DRILL-5117: Summary: Compile error when query a json file with 1000+columns Key: DRILL-5117 URL: https://issues.apache.org/jira/browse/DRILL-5117 Project: Apache Drill Issue Type: Bug Components: Execution - Codegen Affects Versions: 1.8.0 Reporter: Serhii Harnyk Assignee: Serhii Harnyk Fix For: Future Query failed with compile error when we querying a json file with 1000+columns: {noformat} 0: jdbc:drill:zk=local> select * from dfs.`/tmp/tooManyFields.json` limit 1; Error: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB Fragment 0:0 [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] (state=,code=0) {noformat} Stack trace from sqlline.log: {noformat} 2016-12-09 13:43:38,207 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 27b54af4-b41f-0682-e50d-626de4eff68e: select * from dfs.`/tmp/tooManyFields.json` limit 1 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,340 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,341 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,532 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1 2016-12-09 13:43:38,547 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host localhost. Skipping affinity to that host. 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 13ms total, 13.922965ms avg, 13ms max. 2016-12-09 13:43:38,548 [27b54af4-b41f-0682-e50d-626de4eff68e:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 6.956000 μs, Latest start: 6.956000 μs, Average start: 6.956000 μs . 2016-12-09 13:43:38,750 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested AWAITING_ALLOCATION --> RUNNING 2016-12-09 13:43:38,761 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State to report: RUNNING 2016-12-09 13:43:39,375 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] WARN o.a.d.exec.compile.JDKClassCompiler - JDK Java compiler not available - probably you're running Drill with a JRE and not a JDK 2016-12-09 13:43:40,533 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested RUNNING --> FAILED 2016-12-09 13:43:40,550 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 27b54af4-b41f-0682-e50d-626de4eff68e:0:0: State change requested FAILED --> FINISHED 2016-12-09 13:43:40,552 [27b54af4-b41f-0682-e50d-626de4eff68e:frag:0:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: JaninoRuntimeException: Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB Fragment 0:0 [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: JaninoRuntimeException: Code attribute in class "org.apache.drill.exec.test.generated.CopierGen0" grows beyond 64 KB Fragment 0:0 [Error Id: a1306543-4d66-4bb0-b687-5802002833b2 on user515050-pc:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) ~[drill-common-1.8.0.jar:1.8.0] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293) [drill-java-exec-1.8.0.jar:1.8.0] at org.apache.dr
[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/654#discussion_r91351247 --- Diff: contrib/storage-hive/core/src/test/java/org/apache/drill/exec/store/hive/schema/TestColumnListCache.java --- @@ -0,0 +1,78 @@ +/* --- End diff -- Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #673: DRILL-4764: Parquet file with INT_16, etc. logical types n...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/673 @parthchandra, could you please review new changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #654: DRILL-5032: Drill query on hive parquet table failed with ...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/654 @chunhui-shi, I have made some changes. Could you please review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #673: DRILL-4764: Parquet file with INT_16, etc. logical types n...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/673 @parthchandra, I have made some changes and added few test values. Could you please review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #673: DRILL-4764: Parquet file with INT_16, etc. logical ...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/673 DRILL-4764: Parquet file with INT_16, etc. logical types not supporte… …d by simple SELECT You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-4764 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/673.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #673 commit ef8d7a294e1549eb5fbb61da3aad069a478ffca6 Author: Serhii-Harnyk Date: 2016-11-24T13:24:03Z DRILL-4764: Parquet file with INT_16, etc. logical types not supported by simple SELECT --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/654#discussion_r89520154 --- Diff: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java --- @@ -398,12 +398,18 @@ public static void addConfToJob(final JobConf job, final Properties properties) * Wrapper around {@link MetaStoreUtils#getPartitionMetadata(Partition, Table)} which also adds parameters from table * to properties returned by {@link MetaStoreUtils#getPartitionMetadata(Partition, Table)}. * - * @param partition - * @param table - * @return + * @param partition {@link Partition} instance + * @param table {@link Table} instance + * @return properties */ public static Properties getPartitionMetadata(final Partition partition, final Table table) { -final Properties properties = MetaStoreUtils.getPartitionMetadata(partition, table); +final Properties properties; --- End diff -- @chunhui-shi let me clarify. Do you propose something like creating "partition groups" and storing columns in this "groups"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #657: DRILL-5048: AssertionError when case statement is u...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/657 DRILL-5048: AssertionError when case statement is used with timestamp… … and null You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-5048 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/657.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #657 commit 66823553ec9074ee0adbb93f02cc86f41040d81b Author: Serhii-Harnyk Date: 2016-11-17T14:23:58Z DRILL-5048: AssertionError when case statement is used with timestamp and null --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-5048) AssertionError when case statement is used with timestamp and null
Serhii Harnyk created DRILL-5048: Summary: AssertionError when case statement is used with timestamp and null Key: DRILL-5048 URL: https://issues.apache.org/jira/browse/DRILL-5048 Project: Apache Drill Issue Type: Bug Affects Versions: 1.9.0 Reporter: Serhii Harnyk Assignee: Serhii Harnyk Fix For: Future AssertionError when we use case with timestamp and null: {noformat} 0: jdbc:drill:schema=dfs.tmp> SELECT res, CASE res WHEN true THEN CAST('1990-10-10 22:40:50' AS TIMESTAMP) ELSE null END . . . . . . . . . . . . . . > FROM . . . . . . . . . . . . . . > ( . . . . . . . . . . . . . . > SELECT . . . . . . . . . . . . . . > (CASE WHEN (false) THEN null ELSE CAST('1990-10-10 22:40:50' AS TIMESTAMP) END) res . . . . . . . . . . . . . . > FROM (values(1)) foo . . . . . . . . . . . . . . > ) foobar; Error: SYSTEM ERROR: AssertionError: Type mismatch: rowtype of new rel: RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL rowtype of set: RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL [Error Id: b56e0a4d-2f9e-4afd-8c60-5bc2f9d31f8f on centos-01.qa.lab:31010] (state=,code=0) {noformat} Stack trace from drillbit.log {noformat} Caused by: java.lang.AssertionError: Type mismatch: rowtype of new rel: RecordType(TIMESTAMP(0) NOT NULL res, TIMESTAMP(0) EXPR$1) NOT NULL rowtype of set: RecordType(TIMESTAMP(0) res, TIMESTAMP(0) EXPR$1) NOT NULL at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:1696) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.RelSubset.add(RelSubset.java:295) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.RelSet.add(RelSet.java:147) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.VolcanoPlanner.addRelToSet(VolcanoPlanner.java:1818) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1760) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:1017) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1037) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1940) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] at org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:138) ~[calcite-core-1.4.0-drill-r18.jar:1.4.0-drill-r18] ... 16 common frames omitted {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request #654: DRILL-5032: Drill query on hive parquet table faile...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/654 DRILL-5032: Drill query on hive parquet table failed with OutOfMemoryError You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-5032 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/654.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #654 commit 90482d7a1be99293fc3afdf2a297ee08e8831f66 Author: Serhii-Harnyk Date: 2016-10-27T19:20:27Z DRILL-5032 Drill query on hive parquet table failed with OutOfMemoryError: Java heap space --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-5032) Drill query on hive parquet table failed with OutOfMemoryError: Java heap space
Serhii Harnyk created DRILL-5032: Summary: Drill query on hive parquet table failed with OutOfMemoryError: Java heap space Key: DRILL-5032 URL: https://issues.apache.org/jira/browse/DRILL-5032 Project: Apache Drill Issue Type: Bug Components: Functions - Hive Affects Versions: 1.8.0 Reporter: Serhii Harnyk Assignee: Serhii Harnyk Following query on hive parquet table failed with OOM Java heap space: {code} select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12' 2016-08-31 08:02:03,597 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 283938c3-fde8-0fc6-37e1-9a568c7f5913: select distinct(businessdate) from vmdr_trades where trade_date='2016-04-12' 2016-08-31 08:05:58,502 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning class: org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze filter tree: 1 ms 2016-08-31 08:05:58,506 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for partition pruning.Total pruning elapsed time: 3 ms 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning class: org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$2 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze filter tree: 0 ms 2016-08-31 08:05:58,663 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for partition pruning.Total pruning elapsed time: 0 ms 2016-08-31 08:05:58,664 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Beginning partition pruning, pruning class: org.apache.drill.exec.planner.sql.logical.HivePushPartitionFilterIntoScan$1 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - Total elapsed time to build and analyze filter tree: 0 ms 2016-08-31 08:05:58,665 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule - No conditions were found eligible for partition pruning.Total pruning elapsed time: 0 ms 2016-08-31 08:09:42,355 [283938c3-fde8-0fc6-37e1-9a568c7f5913:foreman] ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. Information message: Unable to handle out of memory condition in Foreman. java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_74] at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) ~[na:1.8.0_74] at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) ~[na:1.8.0_74] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) ~[na:1.8.0_74] at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_74] at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457) ~[na:1.8.0_74] at java.lang.StringBuilder.append(StringBuilder.java:166) ~[na:1.8.0_74] at java.lang.StringBuilder.append(StringBuilder.java:76) ~[na:1.8.0_74] at com.google.protobuf.TextFormat$TextGenerator.write(TextFormat.java:538) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat$TextGenerator.print(TextFormat.java:526) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:389) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat.print(TextFormat.java:71) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.TextFormat.printToString(TextFormat.java:118) ~[protobuf-java-2.5.0.jar:na] at com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:106) ~[protobuf-java-2.5.
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 @chunhui-shi Results of performance test | % of nullable records | with fix | master | %(t-t0)/t0 | | 0.001 | 13986,8 | 13752,1 | 1,71% | | 0.1 | 13873,6 | 13081,3 | 6,06% | | 0.5 | 11345,6 | 7552,44 | 50,22% | | 0.7 | 12699,1 | 5753,44 | 120,72% | | 0.99 | 14544,2 | 494 | 2844,17% | So increase of performance depends on % of nullable fields in dataset --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 @chunhui-shi what about second commit? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/581 @adeneche What about fixed milliseconds? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: TO_TIMESTAMP function returns in-correct results
Example of the usage with "s" for second of minute and "S" for fraction of second: VALUES(TO_TIMESTAMP('2015-03-30 20:49:59.10 UTC', '-MM-dd HH:mm:ss.SSS z')) 2016-10-28 16:17 GMT+03:00 Khurram Faraaz : > Thanks Serhii. > > Can you please give me a working example of the usage with "s" for second > of minute and "S" for fraction of second. > > I tried with both those symbols, however Drill 1.9.0 (commit: a29f1e29) > does not honor those symbols when used from within the to_date function. > > On Thu, Oct 27, 2016 at 6:31 PM, Serhii Harnyk > wrote: > > > Hello, Khurram > > > > http://joda-time.sourceforge.net/apidocs/org/joda/time/ > > format/DateTimeFormat.html > > > > s second of minute number55 > > S fraction of second number978 > > > > > > > > 2016-10-27 13:54 GMT+03:00 Khurram Faraaz : > > > > > All, > > > > > > I am on Drill 1.9.0 git commit ID : a29f1e29 on CentOS > > > > > > TO_TIMESTAMP function does not return correct results, note that the > > > minutes, seconds and milliseconds parts of timestamp are incorrect in > the > > > results > > > > > > {noformat} > > > 0: jdbc:drill:schema=dfs.tmp> VALUES(TO_TIMESTAMP('2015-03-30 > > 20:49:59.10 > > > UTC', '-MM-dd HH:mm:ss.s z')); > > > ++ > > > | EXPR$0 | > > > ++ > > > | 2015-03-30 20:49:10.0 | > > > ++ > > > 1 row selected (0.228 seconds) > > > {noformat} > > > > > > {noformat} > > > 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST(TO_TIMESTAMP('2015-03-30 > > > 20:49:59.10 UTC', '-MM-dd HH:mm:ss.s z') AS TIMESTAMP)); > > > ++ > > > | EXPR$0 | > > > ++ > > > | 2015-03-30 20:49:10.0 | > > > ++ > > > 1 row selected (0.265 seconds) > > > {noformat} > > > > > > This case returns correct results, when the same string used above is > > given > > > as input to CAST function, note that minutes mm, seconds ss and > > millisecond > > > s parts are honored > > > > > > {noformat} > > > 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST('2015-03-30 20:49:59.10 UTC' > AS > > > TIMESTAMP)); > > > ++ > > > | EXPR$0 | > > > ++ > > > | 2015-03-30 20:49:59.1 | > > > ++ > > > 1 row selected (0.304 seconds) > > > {noformat} > > > > > > Thanks, > > > Khurram > > > > > >
Re: TO_TIMESTAMP function returns in-correct results
Hello, Khurram http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html s second of minute number55 S fraction of second number978 2016-10-27 13:54 GMT+03:00 Khurram Faraaz : > All, > > I am on Drill 1.9.0 git commit ID : a29f1e29 on CentOS > > TO_TIMESTAMP function does not return correct results, note that the > minutes, seconds and milliseconds parts of timestamp are incorrect in the > results > > {noformat} > 0: jdbc:drill:schema=dfs.tmp> VALUES(TO_TIMESTAMP('2015-03-30 20:49:59.10 > UTC', '-MM-dd HH:mm:ss.s z')); > ++ > | EXPR$0 | > ++ > | 2015-03-30 20:49:10.0 | > ++ > 1 row selected (0.228 seconds) > {noformat} > > {noformat} > 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST(TO_TIMESTAMP('2015-03-30 > 20:49:59.10 UTC', '-MM-dd HH:mm:ss.s z') AS TIMESTAMP)); > ++ > | EXPR$0 | > ++ > | 2015-03-30 20:49:10.0 | > ++ > 1 row selected (0.265 seconds) > {noformat} > > This case returns correct results, when the same string used above is given > as input to CAST function, note that minutes mm, seconds ss and millisecond > s parts are honored > > {noformat} > 0: jdbc:drill:schema=dfs.tmp> VALUES(CAST('2015-03-30 20:49:59.10 UTC' AS > TIMESTAMP)); > ++ > | EXPR$0 | > ++ > | 2015-03-30 20:49:59.1 | > ++ > 1 row selected (0.304 seconds) > {noformat} > > Thanks, > Khurram >
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r84509937 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -337,6 +364,7 @@ private void writeData(MapWriter map, FieldSelection selection, boolean moveForw continue outside; } +nullableFields.remove(fieldName); --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #594: DRILL-4842: SELECT * on JSON data results in NumberFormatE...
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/594 Added fixes according to review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r84509944 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -418,6 +447,7 @@ private void writeDataAllText(MapWriter map, FieldSelection selection, boolean m continue outside; } + nullableFields.remove(fieldName); --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r84507055 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -56,6 +58,12 @@ private final boolean readNumbersAsDouble; /** + * Collection for tracking nullable fields during reading + * and storing them for creating default typed vectors + */ + private final Set nullableFields = Sets.newLinkedHashSet(); --- End diff -- We need to store an order of nullable fields. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r84507516 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -131,14 +139,33 @@ public void ensureAtLeastOneField(ComplexWriter writer) { PathSegment fieldPath = fieldPathList.get(j); if (emptyStatus.get(j)) { if (allTextMode) { - fieldWriter.varChar(fieldPath.getNameSegment().getPath()); + if (checkNullFields(fieldPathList)) { --- End diff -- Added fixes for nested fields with handling "path" of the field. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/594#discussion_r84508112 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java --- @@ -131,14 +139,33 @@ public void ensureAtLeastOneField(ComplexWriter writer) { PathSegment fieldPath = fieldPathList.get(j); if (emptyStatus.get(j)) { if (allTextMode) { - fieldWriter.varChar(fieldPath.getNameSegment().getPath()); + if (checkNullFields(fieldPathList)) { +for (String fieldName : nullableFields) { + fieldWriter.varChar(fieldName); +} + } else { +fieldWriter.varChar(fieldPath.getNameSegment().getPath()); + } } else { - fieldWriter.integer(fieldPath.getNameSegment().getPath()); + if (checkNullFields(fieldPathList)) { --- End diff -- Test added --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill issue #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on the issue: https://github.com/apache/drill/pull/581 @adeneche - miliseconds issue is fixed in last commit --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83871818 --- Diff: logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java --- @@ -0,0 +1,213 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import com.google.common.collect.Sets; +import org.apache.commons.lang3.StringUtils; +import org.apache.drill.common.map.CaseInsensitiveMap; + +import java.util.Comparator; +import java.util.Set; + +public class JodaDateValidator { + + private static final Set ansiValuesForDeleting = Sets.newTreeSet(new LengthDescComparator()); + private static final CaseInsensitiveMap ansiToJodaMap = CaseInsensitiveMap.newTreeMap(new LengthDescComparator()); + + //tokens for deleting + public static final String SUFFIX_SP = "sp"; + public static final String PREFIX_FM = "fm"; + public static final String PREFIX_FX = "fx"; + public static final String PREFIX_TM = "tm"; + + //ansi patterns + public static final String ANSI_FULL_NAME_OF_DAY = "day"; + public static final String ANSI_DAY_OF_YEAR = "ddd"; + public static final String ANSI_DAY_OF_MONTH = "dd"; + public static final String ANSI_DAY_OF_WEEK = "d"; + public static final String ANSI_NAME_OF_MONTH = "month"; + public static final String ANSI_ABR_NAME_OF_MONTH = "mon"; + public static final String ANSI_FULL_ERA_NAME = "ee"; + public static final String ANSI_NAME_OF_DAY = "dy"; + public static final String ANSI_TIME_ZONE_NAME = "tz"; + public static final String ANSI_HOUR_12_NAME = "hh"; + public static final String ANSI_HOUR_12_OTHER_NAME = "hh12"; + public static final String ANSI_HOUR_24_NAME = "hh24"; + public static final String ANSI_MINUTE_OF_HOUR_NAME = "mi"; + public static final String ANSI_SECOND_OF_MINUTE_NAME = "ss"; + public static final String ANSI_MILLISECOND_OF_MINUTE_NAME = "ms"; + public static final String ANSI_WEEK_OF_YEAR = "ww"; + public static final String ANSI_MONTH = "mm"; + public static final String ANSI_HALFDAY_AM = "am"; + public static final String ANSI_HALFDAY_PM = "pm"; + + //jodaTime patterns + public static final String JODA_FULL_NAME_OF_DAY = ""; + public static final String JODA_DAY_OF_YEAR = "D"; + public static final String JODA_DAY_OF_MONTH = "d"; + public static final String JODA_DAY_OF_WEEK = "e"; + public static final String JODA_NAME_OF_MONTH = ""; + public static final String JODA_ABR_NAME_OF_MONTH = "MMM"; + public static final String JODA_FULL_ERA_NAME = "G"; + public static final String JODA_NAME_OF_DAY = "E"; + public static final String JODA_TIME_ZONE_NAME = "TZ"; + public static final String JODA_HOUR_12_NAME = "h"; + public static final String JODA_HOUR_12_OTHER_NAME = "h"; + public static final String JODA_HOUR_24_NAME = "H"; + public static final String JODA_MINUTE_OF_HOUR_NAME = "m"; + public static final String JODA_SECOND_OF_MINUTE_NAME = "s"; + public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "S"; + public static final String JODA_WEEK_OF_YEAR = "w"; + public static final String JODA_MONTH = "MM"; + public static final String JODA_HALFDAY = "aa"; + + static { +ansiToJodaMap.put(ANSI_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY); +ansiToJodaMap.put(ANSI_DAY_OF_YEAR, JODA_DAY_OF_YEAR); +ansiToJodaMap.put(ANSI_DAY_OF_MONTH, JODA_DAY_OF_MONTH); +ansiToJodaMap.put(ANSI_DAY_OF_WEEK
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83871606 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java --- @@ -408,6 +411,12 @@ private LogicalExpression getDrillFunctionFromOptiqCall(RexCall call) { return first; } + } else if (functionName.equals("to_date") || functionName.equals("to_time") || functionName.equals("to_timestamp")) { +// convert ansi date format string to joda according to session option +OptionManager om = this.context.getPlannerSettings().getOptions(); + if(ToDateFormats.valueOf(om.getOption(ExecConstants.TO_DATE_FORMAT).string_val.toUpperCase()).equals(ToDateFormats.ANSI)) { --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83871870 --- Diff: logical/src/main/java/org/apache/drill/common/expression/fn/JodaDateValidator.java --- @@ -0,0 +1,213 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.common.expression.fn; + +import com.google.common.collect.Sets; +import org.apache.commons.lang3.StringUtils; +import org.apache.drill.common.map.CaseInsensitiveMap; + +import java.util.Comparator; +import java.util.Set; + +public class JodaDateValidator { + + private static final Set ansiValuesForDeleting = Sets.newTreeSet(new LengthDescComparator()); + private static final CaseInsensitiveMap ansiToJodaMap = CaseInsensitiveMap.newTreeMap(new LengthDescComparator()); + + //tokens for deleting + public static final String SUFFIX_SP = "sp"; + public static final String PREFIX_FM = "fm"; + public static final String PREFIX_FX = "fx"; + public static final String PREFIX_TM = "tm"; + + //ansi patterns + public static final String ANSI_FULL_NAME_OF_DAY = "day"; + public static final String ANSI_DAY_OF_YEAR = "ddd"; + public static final String ANSI_DAY_OF_MONTH = "dd"; + public static final String ANSI_DAY_OF_WEEK = "d"; + public static final String ANSI_NAME_OF_MONTH = "month"; + public static final String ANSI_ABR_NAME_OF_MONTH = "mon"; + public static final String ANSI_FULL_ERA_NAME = "ee"; + public static final String ANSI_NAME_OF_DAY = "dy"; + public static final String ANSI_TIME_ZONE_NAME = "tz"; + public static final String ANSI_HOUR_12_NAME = "hh"; + public static final String ANSI_HOUR_12_OTHER_NAME = "hh12"; + public static final String ANSI_HOUR_24_NAME = "hh24"; + public static final String ANSI_MINUTE_OF_HOUR_NAME = "mi"; + public static final String ANSI_SECOND_OF_MINUTE_NAME = "ss"; + public static final String ANSI_MILLISECOND_OF_MINUTE_NAME = "ms"; + public static final String ANSI_WEEK_OF_YEAR = "ww"; + public static final String ANSI_MONTH = "mm"; + public static final String ANSI_HALFDAY_AM = "am"; + public static final String ANSI_HALFDAY_PM = "pm"; + + //jodaTime patterns + public static final String JODA_FULL_NAME_OF_DAY = ""; + public static final String JODA_DAY_OF_YEAR = "D"; + public static final String JODA_DAY_OF_MONTH = "d"; + public static final String JODA_DAY_OF_WEEK = "e"; + public static final String JODA_NAME_OF_MONTH = ""; + public static final String JODA_ABR_NAME_OF_MONTH = "MMM"; + public static final String JODA_FULL_ERA_NAME = "G"; + public static final String JODA_NAME_OF_DAY = "E"; + public static final String JODA_TIME_ZONE_NAME = "TZ"; + public static final String JODA_HOUR_12_NAME = "h"; + public static final String JODA_HOUR_12_OTHER_NAME = "h"; + public static final String JODA_HOUR_24_NAME = "H"; + public static final String JODA_MINUTE_OF_HOUR_NAME = "m"; + public static final String JODA_SECOND_OF_MINUTE_NAME = "s"; + public static final String JODA_MILLISECOND_OF_MINUTE_NAME = "S"; + public static final String JODA_WEEK_OF_YEAR = "w"; + public static final String JODA_MONTH = "MM"; + public static final String JODA_HALFDAY = "aa"; + + static { +ansiToJodaMap.put(ANSI_FULL_NAME_OF_DAY, JODA_FULL_NAME_OF_DAY); +ansiToJodaMap.put(ANSI_DAY_OF_YEAR, JODA_DAY_OF_YEAR); +ansiToJodaMap.put(ANSI_DAY_OF_MONTH, JODA_DAY_OF_MONTH); +ansiToJodaMap.put(ANSI_DAY_OF_WEEK
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83871748 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java --- @@ -408,6 +411,12 @@ private LogicalExpression getDrillFunctionFromOptiqCall(RexCall call) { return first; } + } else if (functionName.equals("to_date") || functionName.equals("to_time") || functionName.equals("to_timestamp")) { +// convert ansi date format string to joda according to session option +OptionManager om = this.context.getPlannerSettings().getOptions(); + if(ToDateFormats.valueOf(om.getOption(ExecConstants.TO_DATE_FORMAT).string_val.toUpperCase()).equals(ToDateFormats.ANSI)) { + args.set(1, FunctionCallFactory.createExpression("ansi_to_joda", Arrays.asList(args.get(1; --- End diff -- Yes, it would be two nested ansi_to_joda conversions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83870798 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/AnsiToJoda.java --- @@ -0,0 +1,58 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.exec.expr.fn.impl; + +import io.netty.buffer.DrillBuf; +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.VarCharHolder; + +import javax.inject.Inject; + +/** + * Replaces all ansi patterns to joda equivalents. + */ +@FunctionTemplate(name = "ansi_to_joda", + scope = FunctionTemplate.FunctionScope.SIMPLE, + nulls= FunctionTemplate.NullHandling.NULL_IF_NULL) --- End diff -- Fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83871584 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java --- @@ -408,6 +411,12 @@ private LogicalExpression getDrillFunctionFromOptiqCall(RexCall call) { return first; } + } else if (functionName.equals("to_date") || functionName.equals("to_time") || functionName.equals("to_timestamp")) { --- End diff -- No. Variable functionName has been already lowercased. It's for all functions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/581#discussion_r83871296 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/AnsiToJoda.java --- @@ -0,0 +1,58 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.exec.expr.fn.impl; + +import io.netty.buffer.DrillBuf; +import org.apache.drill.exec.expr.DrillSimpleFunc; +import org.apache.drill.exec.expr.annotations.FunctionTemplate; +import org.apache.drill.exec.expr.annotations.Output; +import org.apache.drill.exec.expr.annotations.Param; +import org.apache.drill.exec.expr.holders.VarCharHolder; + +import javax.inject.Inject; + +/** + * Replaces all ansi patterns to joda equivalents. + */ +@FunctionTemplate(name = "ansi_to_joda", + scope = FunctionTemplate.FunctionScope.SIMPLE, + nulls= FunctionTemplate.NullHandling.NULL_IF_NULL) +public class AnsiToJoda implements DrillSimpleFunc { + + @Param + VarCharHolder in; + + @Output + VarCharHolder out; + + @Inject + DrillBuf buffer; + + @Override + public void setup() { + } + + @Override + public void eval() { +String pattern = org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(in.start, in.end, in.buffer); --- End diff -- Datetime patterns validation isn't a simple task. For example JODA Time has no validation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #598: DRILL-4906 CASE Expression with constant generates ...
Github user Serhii-Harnyk commented on a diff in the pull request: https://github.com/apache/drill/pull/598#discussion_r81115837 --- Diff: exec/java-exec/src/test/java/org/apache/drill/exec/planner/logical/TestCaseNullableTypes.java --- @@ -0,0 +1,107 @@ +/* +* Licensed to the Apache Software Foundation (ASF) under one or more +* contributor license agreements. See the NOTICE file distributed with +* this work for additional information regarding copyright ownership. +* The ASF licenses this file to you under the Apache License, Version 2.0 +* (the "License"); you may not use this file except in compliance with +* the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +*/ +package org.apache.drill.exec.planner.logical; + +import org.apache.drill.BaseTestQuery; +import org.junit.Test; + +/** + * DRILL-4906 + * Tests for handling nullable types in CASE function + */ +public class TestCaseNullableTypes extends BaseTestQuery { + + @Test + public void testCaseNullableTypesInt() throws Exception { +testBuilder() +.sqlQuery("select (case when (false) then null else 1 end) res1 from (values(1))") +.unOrdered() +.baselineColumns("res1") +.baselineValues(1) +.go(); + } + + @Test + public void testCaseNullableTypesVarchar() throws Exception { --- End diff -- Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #598: DRILL-4906 CASE Expression with constant generates ...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/598 DRILL-4906 CASE Expression with constant generates class exception You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-4906 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/598.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #598 commit 46f0df7cf89fff8e7d8cf9a21a810e1b4e292bce Author: Serhii-Harnyk Date: 2016-09-22T12:06:10Z DRILL-4906 CASE Expression with constant generates class exception --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (DRILL-4906) CASE Expression with constant generates class exception
Serhii Harnyk created DRILL-4906: Summary: CASE Expression with constant generates class exception Key: DRILL-4906 URL: https://issues.apache.org/jira/browse/DRILL-4906 Project: Apache Drill Issue Type: Bug Components: SQL Parser Affects Versions: 1.8.0, 1.6.0 Reporter: Serhii Harnyk Assignee: Serhii Harnyk Fix For: 1.9.0 How to reproduce: select (case when (true) then 1 end) from (values(1)); Error Error: SYSTEM ERROR: ClassCastException: org.apache.drill.exec.expr.holders.NullableVarCharHolder cannot be cast to org.apache.drill.exec.expr.holders.VarCharHolder -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] drill pull request #594: DRILL-4842: SELECT * on JSON data results in Number...
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/594 DRILL-4842: SELECT * on JSON data results in NumberFormatException You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-4842 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/594.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #594 commit 190a69a65ad8c144b164c2acacf9718a0ecb3768 Author: Serhii-Harnyk Date: 2016-09-08T18:11:37Z DRILL-4842: SELECT * on JSON data results in NumberFormatException --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] drill pull request #581: DRILL-4864: Add ANSI format for date/time functions
GitHub user Serhii-Harnyk opened a pull request: https://github.com/apache/drill/pull/581 DRILL-4864: Add ANSI format for date/time functions You can merge this pull request into a Git repository by running: $ git pull https://github.com/Serhii-Harnyk/drill DRILL-4864 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/581.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #581 commit e580e38290de4e3a4f7510811e3c5a956c5d7594 Author: Serhii-Harnyk Date: 2016-09-01T14:48:02Z DRILL-4864: Add ANSI format for date/time functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---