[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087971#comment-14087971
]
Jason Dere commented on HIVE-7421:
--
+1 if tests pass
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: HIVE-7421.1.patch, TestWithORC.zip, fail_47.sql,
> fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087218#comment-14087218
]
Matt McCline commented on HIVE-7421:
Make VectorUDFDateString use the same date parsing and formatting as
GenericUDFDate, which is its non-vectorized counterpart.
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: HIVE-7421.1.patch, TestWithORC.zip, fail_47.sql,
> fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087101#comment-14087101
]
Matt McCline commented on HIVE-7421:
*Postgres* results (using necessarily modified query):
{code}
mmccline=# SELECT order_priority AS none_order_priority_nk FROM testv1_Staples
WHERE ((CONCAT(TO_DATE(order_date_,'-MM-DD'), ' 00:00:00') = '1997-01-01
00:00:00' OR CONCAT(TO_DATE(order_date_,'-MM-DD'), ' 00:00:00') =
'1997-01-03 00:00:00') AND (TO_DATE(order_date_,'-MM-DD') = '1997-01-01' OR
TO_DATE(order_date_,'-MM-DD') = '1997-01-03')) GROUP BY order_priority;
none_order_priority_nk
4-NOT SPECIFIED
1-URGENT
2-HIGH
3-MEDIUM
5-LOW
(5 rows)
{code}
Non-vectorized, M/R or Tez:
{code}
SELECT `Staples`.`order_priority` AS `none_order_priority_nk` FROM
`default`.`testv1_Staples` `Staples` WHERE
((CONCAT(TO_DATE(`Staples`.`order_date_`), ' 00:00:00') = '1997-01-01 00:00:00'
OR CONCAT(TO_DATE(`Staples`.`order_date_`), ' 00:00:00') = '1997-01-03
00:00:00') AND (TO_DATE(`Staples`.`order_date_`) = '1997-01-01' OR
TO_DATE(`Staples`.`order_date_`) = '1997-01-03')) GROUP BY
`Staples`.`order_priority` ;
1-URGENT
2-HIGH
3-MEDIUM
4-NOT SPECIFIED
5-LOW
{code}
*NO RESULTS* when Vectorized.
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069219#comment-14069219
]
Matt McCline commented on HIVE-7421:
Yes, I noticed the query catches the exceptions, suppresses it, and doesn't
"fail", except it returns no rows. But the non-vectorized query does return
rows.
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067811#comment-14067811
]
Jitendra Nath Pandey commented on HIVE-7421:
bq. Is this an error in the query or in our execution of it?
It is possible that the data contains a string that cannot be parsed as a
DATE and those strings are parsed into null value. VectorUDFDateString actually
catches the IllegalArgumentException and returns null, therefore the query
should not fail.
Is the query failing in vectorized path? Or is it returning wrong result?
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067803#comment-14067803
]
Matt McCline commented on HIVE-7421:
Non-vectorized query produces output.
{code}
1-URGENT
2-HIGH
3-MEDIUM
4-NOT SPECIFIED
5-LOW
{code}
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067801#comment-14067801
]
Matt McCline commented on HIVE-7421:
The dates being passed to VectorUDFDateString which calls Date.valueOf include
the time 00:00:00, which isn't allowed...
{code}
14/07/19 21:29:35 INFO vector.VectorGroupByOperator: VectorUDFDateString string
passed to Date.valueOf = '2002-12-31 00:00:00'
{code}
Is this an error in the query or in our execution of it?
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067764#comment-14067764
]
Matt McCline commented on HIVE-7421:
Made this change.
Now there are exceptions being thrown and caught below in the same call
sequence.
{code}
java.lang.IllegalArgumentException
at java.sql.Date.valueOf(Date.java:138)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateString$1.evaluate(VectorUDFDateString.java:38)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.StringUnaryUDF.evaluate(StringUnaryUDF.java:101)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:50)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringColEqualStringScalar.evaluate(FilterStringColEqualStringScalar.java:48)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprOrExpr.evaluate(FilterExprOrExpr.java:87)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:37)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.processOp(VectorFilterOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
{code}
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLo
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066115#comment-14066115
]
Jitendra Nath Pandey commented on HIVE-7421:
StringConcatColScalar is not making a recursive call to evaluate its children.
I think that is causing the columns to show up with nulls.
StringConcatColScalar#evaluate should have following code at the beginning.
{code}
if (childExpressions != null) {
super.evaluateChildren(batch);
}
{code}
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066093#comment-14066093
]
Matt McCline commented on HIVE-7421:
Here is the explain output for query 47 with SPECIAL annotation showing the
VectorExpression(s):
{code}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: staples
Statistics: Num rows: 54860 Data size: 158216240 Basic stats:
COMPLETE Column stats: NONE
Filter Operator
predicate: (((concat(to_date(order_date_), ' 00:00:00') =
'1997-01-01 00:00:00') or (concat(to_date(order_date_), ' 00:00:00') =
'1997-01-03 00:00:00')) and ((to_date(order_date_) = '1997-01-01') or
(to_date(order_date_) = '1997-01-03'))) (type: boolean)
Statistics: Num rows: 54860 Data size: 158216240 Basic stats:
COMPLETE Column stats: NONE
vector filter expressions:
FilterExprAndExpr[-1](FilterExprOrExpr[-1](FilterStringColEqualStringScalar[-1](StringConcatColScalar[51](VectorUDFDateString[50]))
FilterStringColEqualStringScalar[-1](StringConcatColScalar[51](VectorUDFDateString[50])))
FilterExprOrExpr[-1](FilterStringColEqualStringScalar[-1](VectorUDFDateString[50])
FilterStringColEqualStringScalar[-1](VectorUDFDateString[50])))
Select Operator
expressions: order_priority (type: string)
outputColumnNames: order_priority
Statistics: Num rows: 54860 Data size: 158216240 Basic stats:
COMPLETE Column stats: NONE
vector select expressions: IdentityExpression[2]
Group By Operator
keys: order_priority (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 54860 Data size: 158216240 Basic stats:
COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 54860 Data size: 158216240 Basic
stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 27430 Data size: 79108120 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 27430 Data size: 79108120 Basic stats:
COMPLETE Column stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 27430 Data size: 79108120 Basic stats:
COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
{code}
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
> Attachments: TestWithORC.zip, fail_47.sql, fail_62.sql, fail_932.sql
>
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(
[jira] [Commented] (HIVE-7421) Null pointer exception involving ql.exec.vector.expressions.StringConcatColScalar.evaluate
[
https://issues.apache.org/jira/browse/HIVE-7421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062925#comment-14062925
]
Matt McCline commented on HIVE-7421:
Repro:
0) Download the attached TestWithORC.zip file
1) Substitute /tmp/TestV1 in HiveORCCreateTable.sql for a better local
location, if desired
3) Run command to create schema and load data: hive -f HiveORCCreateTable.sql
4) Run Hive:
set hive.vectorized.execution.enabled=true;
source fail_47.sql;
(Or, fail_62.sql or fail_932.sql)
> Null pointer exception involving
> ql.exec.vector.expressions.StringConcatColScalar.evaluate
> --
>
> Key: HIVE-7421
> URL: https://issues.apache.org/jira/browse/HIVE-7421
> Project: Hive
> Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>
> One of several found by Raj Bains.
> M/R or Tez.
> {code}
> set hive.vectorized.execution.enabled=true;
> {code}
> Seems very similar to https://issues.apache.org/jira/browse/HIVE-6649
> Query:
> {code}
> SELECT FLOOR((7 + DATEDIFF(`Staples`.`order_date_`,
> CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01 00:00:00'))
> +pmod(8 + pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING),
> '-01-01 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) AS `wk_order_date_ok`,
> SUM(`Staples`.`sales_total`) AS `sum_sales_total_ok` FROM
> `default`.`testv1_Staples` `Staples` GROUP BY FLOOR((7 +
> DATEDIFF(`Staples`.`order_date_`, CONCAT(CAST(YEAR(`Staples`.`order_date_`)
> AS STRING), '-01-01 00:00:00')) +pmod(8 +
> pmod(datediff(CONCAT(CAST(YEAR(`Staples`.`order_date_`) AS STRING), '-01-01
> 00:00:00'), '1995-01-01'), 7) - 2, 7) ) / 7) ;
> {code}
> Stack trace:
> {code}
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setConcat(BytesColumnVector.java:190)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.StringConcatColScalar.evaluate(StringConcatColScalar.java:78)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorUDFDateDiffColCol.evaluate(VectorUDFDateDiffColCol.java:59)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongScalarAddLongColumn.evaluate(LongScalarAddLongColumn.java:65)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.LongColAddLongColumn.evaluate(LongColAddLongColumn.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.LongColDivideLongScalar.evaluate(LongColDivideLongScalar.java:52)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:112)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FuncFloorDoubleToLong.evaluate(FuncFloorDoubleToLong.java:47)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.evaluateBatch(VectorHashKeyWrapperBatch.java:147)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:289)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:711)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
