[jira] [Created] (HIVE-11804) Different describe formatted behavior depending on whether the table name is qualified with database name or not
Mark Grover created HIVE-11804:
--
Summary: Different describe formatted behavior depending on
whether the table name is qualified with database name or not
Key: HIVE-11804
URL: https://issues.apache.org/jira/browse/HIVE-11804
Project: Hive
Issue Type: Bug
Components: Metastore
Reporter: Mark Grover
I have a simple text file based managed table on HDFS:
{quote}
show create table src;
+---+--+
|createtab_stmt
|
+---+--+
| CREATE TABLE `src`(
|
| `first` string,
|
| `word` string)
|
| PARTITIONED BY (
|
| `length` int)
|
| ROW FORMAT SERDE
|
| 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
|
| STORED AS INPUTFORMAT
|
| 'org.apache.hadoop.mapred.TextInputFormat'
|
| OUTPUTFORMAT
|
| 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
|
| LOCATION
|
| 'hdfs://name-node:8020/user/hive/warehouse/my.db/src' |
| TBLPROPERTIES (
|
| 'transient_lastDdlTime'='1441921577')
|
+---+--+
{quote}
The describe formatted with the database name returns:
{quote}
describe formatted my.src first partition(length=1);
+-+---+---+---++-+--+--++-+--+--+
|col_name | data_type | min |
max | num_nulls | distinct_count | avg_col_len | max_col_len | num_trues
| num_falses | comment |
+-+---+---+---++-+--+--++-+--+--+
| # col_name | data_type | comment |
| NULL | NULL| NULL | NULL | NULL |
NULL| NULL |
| | NULL | NULL |
NULL | NULL | NULL| NULL | NULL | NULL
| NULL| NULL |
| first | string| from deserializer |
NULL | NULL | NULL| NULL | NULL | NULL
| NULL| NULL |
+-+---+---+---++-+--+--++-+--+--+
{quote}
while without it returns:
{quote}
describe formatted src first partition(length=1);
+---+---+---+--+
| col_name| data_type
|comment|
+---+---+---+--+
| # col_name| data_type
| comment |
| | NULL
| NULL |
| first | string
| |
| word | string
| |
| | NULL
| NULL |
| # Partition Information | NULL
| NULL |
| # col_name| data_type
| comment |
| | NULL
[jira] [Created] (HIVE-10041) Set defaults for HBASE_HOME in a smarter way
Mark Grover created HIVE-10041: -- Summary: Set defaults for HBASE_HOME in a smarter way Key: HIVE-10041 URL: https://issues.apache.org/jira/browse/HIVE-10041 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 1.1.0 Reporter: Mark Grover Assignee: Mark Grover Fix For: 1.1.0 Similar to SQOOP-2145, hcat binary script doesn't do smart detection of HBASE_HOME. It assumes it's always located in /usr/lib/hbase and if not HBASE_HOME variable needs to be exported. The reason is often times people have tarballs like ~/hive-hcatalog, ~/hbase, etc. and it would be good to have their hcat scripts work out of the book. This doesn't regress anything because it only sets the HBASE_HOME if it's not already set and the directory it's setting it to exists and is valid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9818) Print error messages on stderr instead of stdout
Mark Grover created HIVE-9818:
-
Summary: Print error messages on stderr instead of stdout
Key: HIVE-9818
URL: https://issues.apache.org/jira/browse/HIVE-9818
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 1.0.0
Reporter: Mark Grover
hcat script (see
[here|https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L103], for
example) prints error messages via echo which go on stdout the way they are
written. This is bad because downstream projects using it receive error
messages as expected output instead of actually receiving nothing (see
SQOOP-2147 as example).
We need to put error messages on stdout. Something like:
{code}
>&2 echo "This is an error message"
{code}
instead of a simple echo that we have been using so far.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9000) LAST_VALUE Window function returns wrong results
[
https://issues.apache.org/jira/browse/HIVE-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-9000:
--
Description:
LAST_VALUE Windowing function has been returning bad results, as far as I can
tell from day 1.
And, it seems like the tests are also asserting that LAST_VALUE gives the wrong
result.
Here's the test output:
https://github.com/apache/hive/blob/branch-0.14/ql/src/test/results/clientpositive/windowing_navfn.q.out#L587
The query is:
{code}
select t, s, i, last_value(i) over (partition by t order by s)
{code}
The result is:
{code}
t si last_value(i)
---
10 oscar allen 65662 65662
10 oscar carson65549 65549
{code}
{{LAST_VALUE( i )}} should have returned 65549 in both records, instead it
simply ends up returning i.
Another way you can make sure LAST_VALUE is bad is to verify it's result
against LEAD(i,1) over (partition by t order by s). LAST_VALUE being last value
should always be more (in terms of the specified 'order by s') than the lead by
1. While this doesn't directly apply to the above query, if the result set had
more rows, you would clearly see records where lead is higher than last_value
which is semantically incorrect.
was:
LAST_VALUE Windowing function has been returning bad results, as far as I can
tell from day 1.
And, it seems like the tests are also asserting that LAST_VALUE gives the wrong
result.
Here's the test output:
https://github.com/apache/hive/blob/branch-0.14/ql/src/test/results/clientpositive/windowing_navfn.q.out#L587
The query is:
{code}
select t, s, i, last_value(i) over (partition by t order by s)
{code}
The result is:
{code}
t si last_value(i)
---
10 oscar allen 65662 65662
10 oscar carson65549 65549
{code}
LAST_VALUE(i) should have returned 65549 in both records, instead it simply
ends up returning i.
Another way you can make sure LAST_VALUE is bad is to verify it's result
against LEAD(i,1) over (partition by t order by s). LAST_VALUE being last value
should always be more (in terms of the specified 'order by s') than the lead by
1. While this doesn't directly apply to the above query, if the result set had
more rows, you would clearly see records where lead is higher than last_value
which is semantically incorrect.
> LAST_VALUE Window function returns wrong results
>
>
> Key: HIVE-9000
> URL: https://issues.apache.org/jira/browse/HIVE-9000
> Project: Hive
> Issue Type: Bug
> Components: PTF-Windowing
>Affects Versions: 0.13.1
>Reporter: Mark Grover
>Priority: Critical
> Fix For: 0.14.1
>
>
> LAST_VALUE Windowing function has been returning bad results, as far as I can
> tell from day 1.
> And, it seems like the tests are also asserting that LAST_VALUE gives the
> wrong result.
> Here's the test output:
> https://github.com/apache/hive/blob/branch-0.14/ql/src/test/results/clientpositive/windowing_navfn.q.out#L587
> The query is:
> {code}
> select t, s, i, last_value(i) over (partition by t order by s)
> {code}
> The result is:
> {code}
> t si last_value(i)
> ---
> 10oscar allen 65662 65662
> 10oscar carson65549 65549
> {code}
> {{LAST_VALUE( i )}} should have returned 65549 in both records, instead it
> simply ends up returning i.
> Another way you can make sure LAST_VALUE is bad is to verify it's result
> against LEAD(i,1) over (partition by t order by s). LAST_VALUE being last
> value should always be more (in terms of the specified 'order by s') than the
> lead by 1. While this doesn't directly apply to the above query, if the
> result set had more rows, you would clearly see records where lead is higher
> than last_value which is semantically incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9000) LAST_VALUE Window function returns wrong results
Mark Grover created HIVE-9000:
-
Summary: LAST_VALUE Window function returns wrong results
Key: HIVE-9000
URL: https://issues.apache.org/jira/browse/HIVE-9000
Project: Hive
Issue Type: Bug
Components: PTF-Windowing
Affects Versions: 0.13.1
Reporter: Mark Grover
Priority: Critical
Fix For: 0.14.1
LAST_VALUE Windowing function has been returning bad results, as far as I can
tell from day 1.
And, it seems like the tests are also asserting that LAST_VALUE gives the wrong
result.
Here's the test output:
https://github.com/apache/hive/blob/branch-0.14/ql/src/test/results/clientpositive/windowing_navfn.q.out#L587
The query is:
{code}
select t, s, i, last_value(i) over (partition by t order by s)
{code}
The result is:
{code}
t si last_value(i)
---
10 oscar allen 65662 65662
10 oscar carson65549 65549
{code}
LAST_VALUE(i) should have returned 65549 in both records, instead it simply
ends up returning i.
Another way you can make sure LAST_VALUE is bad is to verify it's result
against LEAD(i,1) over (partition by t order by s). LAST_VALUE being last value
should always be more (in terms of the specified 'order by s') than the lead by
1. While this doesn't directly apply to the above query, if the result set had
more rows, you would clearly see records where lead is higher than last_value
which is semantically incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6593) Create a maven assembly for hive-jdbc
[ https://issues.apache.org/jira/browse/HIVE-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924974#comment-13924974 ] Mark Grover commented on HIVE-6593: --- Thanks Szehon for taking this up and everyone for their input. Cos or I can review the patch. > Create a maven assembly for hive-jdbc > - > > Key: HIVE-6593 > URL: https://issues.apache.org/jira/browse/HIVE-6593 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Affects Versions: 0.12.0 >Reporter: Mark Grover >Assignee: Szehon Ho > Attachments: HIVE-6593.patch > > > Currently in Apache Bigtop we bundle and distribute Hive. In particular, for > users to not have to install the entirety of Hive on machines that are just > jdbc clients, we have a special package which is a subset of hive, called > hive-jdbc that bundles only the jdbc driver jar and it's dependencies. > However, because Hive doesn't have an assembly for the jdbc jar, we have to > hack and hardcode the list of jdbc jars and it's dependencies: > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hive/SPECS/hive.spec#L361 > As Hive moves to Maven, it would be pretty fantastic if Hive could leverage > the maven-assembly-plugin and generate a .tar.gz assembly for what's required > for jdbc gateway machines. That we can simply take that distribution and > build a jdbc package from it without having to hard code jar names and > dependencies. That would make the process much less error prone. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HIVE-6593) Create a maven assembly for hive-jdbc
Mark Grover created HIVE-6593: - Summary: Create a maven assembly for hive-jdbc Key: HIVE-6593 URL: https://issues.apache.org/jira/browse/HIVE-6593 Project: Hive Issue Type: Improvement Components: Build Infrastructure Affects Versions: 0.12.0 Reporter: Mark Grover Currently in Apache Bigtop we bundle and distribute Hive. In particular, for users to not have to install the entirety of Hive on machines that are just jdbc clients, we have a special package which is a subset of hive, called hive-jdbc that bundles only the jdbc driver jar and it's dependencies. However, because Hive doesn't have an assembly for the jdbc jar, we have to hack and hardcode the list of jdbc jars and it's dependencies: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hive/SPECS/hive.spec#L361 As Hive moves to Maven, it would be pretty fantastic if Hive could leverage the maven-assembly-plugin and generate a .tar.gz assembly for what's required for jdbc gateway machines. That we can simply take that distribution and build a jdbc package from it without having to hard code jar names and dependencies. That would make the process much less error prone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-3819) Creating a table on Hive without Hadoop daemons running returns a misleading error
[
https://issues.apache.org/jira/browse/HIVE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817348#comment-13817348
]
Mark Grover commented on HIVE-3819:
---
Sorry, Xuefu, I haven't had the time. If you can't reproduce this, please go
ahead and mark this as Cant' reproduce.
Thanks for checking!
> Creating a table on Hive without Hadoop daemons running returns a misleading
> error
> --
>
> Key: HIVE-3819
> URL: https://issues.apache.org/jira/browse/HIVE-3819
> Project: Hive
> Issue Type: Bug
> Components: CLI, Metastore
>Reporter: Mark Grover
>Assignee: Xuefu Zhang
>
> I was running hive without running the underlying hadoop daemon's running.
> Hadoop was configured to run in pseudo-distributed mode. However, when I
> tried to create a hive table, I got this rather misleading error:
> {code}
> FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> {code}
> We should look into making this error message less misleading (more about
> hadoop daemons not running instead of metastore client not being
> instantiable).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HIVE-3844) Unix timestamps don't seem to be read correctly from HDFS as Timestamp column
[
https://issues.apache.org/jira/browse/HIVE-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover reassigned HIVE-3844:
-
Assignee: Venki Korukanti (was: Mark Grover)
Venki,
I am not. Assigned it to you.
> Unix timestamps don't seem to be read correctly from HDFS as Timestamp column
> -
>
> Key: HIVE-3844
> URL: https://issues.apache.org/jira/browse/HIVE-3844
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 0.8.0
>Reporter: Mark Grover
>Assignee: Venki Korukanti
>
> Serega Shepak pointed out that something like
> {code}
> select cast(date_occurrence as timestamp) from xvlr_data limit 10
> {code}
> where date_occurrence has BIGINT type (timestamp in milliseconds) works. But
> it doesn't work if the declared type is TIMESTAMP on column. The data in the
> date_occurence column in unix timestamp in millis.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-3819) Creating a table on Hive without Hadoop daemons running returns a misleading error
[
https://issues.apache.org/jira/browse/HIVE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800064#comment-13800064
]
Mark Grover commented on HIVE-3819:
---
Stupid autocorrect! I meant to say let me try to reproduce it:-)
> Creating a table on Hive without Hadoop daemons running returns a misleading
> error
> --
>
> Key: HIVE-3819
> URL: https://issues.apache.org/jira/browse/HIVE-3819
> Project: Hive
> Issue Type: Bug
> Components: CLI, Metastore
>Reporter: Mark Grover
>Assignee: Xuefu Zhang
>
> I was running hive without running the underlying hadoop daemon's running.
> Hadoop was configured to run in pseudo-distributed mode. However, when I
> tried to create a hive table, I got this rather misleading error:
> {code}
> FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> {code}
> We should look into making this error message less misleading (more about
> hadoop daemons not running instead of metastore client not being
> instantiable).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-3819) Creating a table on Hive without Hadoop daemons running returns a misleading error
[
https://issues.apache.org/jira/browse/HIVE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800063#comment-13800063
]
Mark Grover commented on HIVE-3819:
---
Let merry to reproduce it.
> Creating a table on Hive without Hadoop daemons running returns a misleading
> error
> --
>
> Key: HIVE-3819
> URL: https://issues.apache.org/jira/browse/HIVE-3819
> Project: Hive
> Issue Type: Bug
> Components: CLI, Metastore
>Reporter: Mark Grover
>Assignee: Xuefu Zhang
>
> I was running hive without running the underlying hadoop daemon's running.
> Hadoop was configured to run in pseudo-distributed mode. However, when I
> tried to create a hive table, I got this rather misleading error:
> {code}
> FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> {code}
> We should look into making this error message less misleading (more about
> hadoop daemons not running instead of metastore client not being
> instantiable).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HIVE-5534) HCatalog shell scripts have non-executable permissions
[
https://issues.apache.org/jira/browse/HIVE-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-5534:
--
Priority: Critical (was: Major)
Summary: HCatalog shell scripts have non-executable permissions (was:
Webhcatalog shell scripts have non-executable permissions)
> HCatalog shell scripts have non-executable permissions
> --
>
> Key: HIVE-5534
> URL: https://issues.apache.org/jira/browse/HIVE-5534
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Mark Grover
>Priority: Critical
> Fix For: 0.12.0
>
>
> While playing with the hive 0.12 rc1, I noticed that the script for starting
> and stopping webhcatalog server don't have the correct permissions.
> {code}
> localhost:hive-0.12.0 mgrover$ find . -name *webhcat*.sh | grep -v './src/' |
> xargs ls -lrt
> -rw-r--r-- 1 mgrover staff 7141 Oct 9 18:50
> ./hcatalog/sbin/webhcat_server.sh
> -rw-r--r-- 1 mgrover staff 4243 Oct 9 18:50
> ./hcatalog/sbin/webhcat_config.sh
> {code}
> This means that we can't start or stop webhcat server using the tarball out
> of the box.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5534) Webhcatalog shell scripts have non-executable permissions
[
https://issues.apache.org/jira/browse/HIVE-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794386#comment-13794386
]
Mark Grover commented on HIVE-5534:
---
Poking more seems like this a problem with all hcatalog scripts so starting
hcatalog server is affected as well:
{code}
-rw-r--r-- 1 mgrover staff 2048 Oct 9 18:50
./hcatalog/share/hcatalog/scripts/hcat_server_stop.sh
-rw-r--r-- 1 mgrover staff 3003 Oct 9 18:50
./hcatalog/share/hcatalog/scripts/hcat_server_start.sh
-rw-r--r-- 1 mgrover staff 4698 Oct 9 18:50
./hcatalog/share/hcatalog/scripts/hcat_server_install.sh
-rw-r--r-- 1 mgrover staff 7141 Oct 9 18:50
./hcatalog/sbin/webhcat_server.sh
-rw-r--r-- 1 mgrover staff 4243 Oct 9 18:50
./hcatalog/sbin/webhcat_config.sh
-rw-r--r-- 1 mgrover staff 10013 Oct 9 18:50
./hcatalog/sbin/update-hcatalog-env.sh
-rw-r--r-- 1 mgrover staff 4540 Oct 9 18:50 ./hcatalog/sbin/hcat_server.sh
-rw-r--r-- 1 mgrover staff 2543 Oct 9 18:50
./hcatalog/libexec/hcat-config.sh
{code}
I update the title of the JIRA to reflect the above.
> Webhcatalog shell scripts have non-executable permissions
> -
>
> Key: HIVE-5534
> URL: https://issues.apache.org/jira/browse/HIVE-5534
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Mark Grover
> Fix For: 0.12.0
>
>
> While playing with the hive 0.12 rc1, I noticed that the script for starting
> and stopping webhcatalog server don't have the correct permissions.
> {code}
> localhost:hive-0.12.0 mgrover$ find . -name *webhcat*.sh | grep -v './src/' |
> xargs ls -lrt
> -rw-r--r-- 1 mgrover staff 7141 Oct 9 18:50
> ./hcatalog/sbin/webhcat_server.sh
> -rw-r--r-- 1 mgrover staff 4243 Oct 9 18:50
> ./hcatalog/sbin/webhcat_config.sh
> {code}
> This means that we can't start or stop webhcat server using the tarball out
> of the box.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HIVE-5534) Webhcatalog shell scripts have non-executable permissions
Mark Grover created HIVE-5534:
-
Summary: Webhcatalog shell scripts have non-executable permissions
Key: HIVE-5534
URL: https://issues.apache.org/jira/browse/HIVE-5534
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.11.0, 0.12.0
Reporter: Mark Grover
Fix For: 0.12.0
While playing with the hive 0.12 rc1, I noticed that the script for starting
and stopping webhcatalog server don't have the correct permissions.
{code}
localhost:hive-0.12.0 mgrover$ find . -name *webhcat*.sh | grep -v './src/' |
xargs ls -lrt
-rw-r--r-- 1 mgrover staff 7141 Oct 9 18:50
./hcatalog/sbin/webhcat_server.sh
-rw-r--r-- 1 mgrover staff 4243 Oct 9 18:50
./hcatalog/sbin/webhcat_config.sh
{code}
This means that we can't start or stop webhcat server using the tarball out of
the box.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-3976) Support specifying scale and precision with Hive decimal type
[
https://issues.apache.org/jira/browse/HIVE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765119#comment-13765119
]
Mark Grover commented on HIVE-3976:
---
Hi Xuefu,
Thanks for posting the document. Overall, looks good! It's great that you are
working on this.
One minor question, in the document you mention "Specifically, for +/-, the
result scale is s1 + s2, and for multiplication, max(s1, s2)". Did you mean the
other way around (max for +/-, s1+s2 for multiplication)?
In the error handling case, you mention rounding instead of using null. That
definitely seems like a plausible idea. However, I think it may be a slippery
slope. For example, we may be using a rounding policy here by default. This
policy may not work for all users and then we may have to expose
configuration/table specific options to specify such rounding policies. There
might be value in masking the values that don't conform as nulls and putting
the onus on the users to explicitly round them to conform to the column type.
However, if mysql or another popular database has set a precedent, what you are
suggesting may be ok.
Big +1 on changing arithmetic operation UDFs to GenericUDFs like you and Jason
already suggested.
> Support specifying scale and precision with Hive decimal type
> -
>
> Key: HIVE-3976
> URL: https://issues.apache.org/jira/browse/HIVE-3976
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, Types
>Reporter: Mark Grover
>Assignee: Xuefu Zhang
> Attachments: remove_prec_scale.diff
>
>
> HIVE-2693 introduced support for Decimal datatype in Hive. However, the
> current implementation has unlimited precision and provides no way to specify
> precision and scale when creating the table.
> For example, MySQL allows users to specify scale and precision of the decimal
> datatype when creating the table:
> {code}
> CREATE TABLE numbers (a DECIMAL(20,2));
> {code}
> Hive should support something similar too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in exec.Utilities
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762057#comment-13762057 ] Mark Grover commented on HIVE-4003: --- Thank you! > NullPointerException in exec.Utilities > -- > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover >Priority: Blocker > Fix For: 0.12.0, 0.13.0 > > Attachments: HIVE-4003.patch, HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762017#comment-13762017 ] Mark Grover commented on HIVE-4003: --- [~appodictic] or [~brocknoland] would one of you mind committing this? > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Attachments: HIVE-4003.patch, HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-4003: -- Attachment: (was: HIVE-4003.2.patch) > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Attachments: HIVE-4003.patch, HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-4003: -- Attachment: HIVE-4003.2.patch Uploaded a new rebased patch. > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Attachments: HIVE-4003.patch, HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-4003: -- Status: Patch Available (was: Open) > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Attachments: HIVE-4003.patch, HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-4003: -- Attachment: HIVE-4003.patch I am guessing I have to keep the name of the patch same, so tests can be run. Correcting the name to be HIVE-4003.patch now. > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Attachments: HIVE-4003.patch, HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738355#comment-13738355 ] Mark Grover commented on HIVE-4003: --- Thanks, Brock, for taking a look. I will rebase this. > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Attachments: HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
[
https://issues.apache.org/jira/browse/HIVE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737106#comment-13737106
]
Mark Grover commented on HIVE-4388:
---
Brock, thanks for looking into this.
I was reviewing the patch and saw that you have several references to
{{getFamilyMap()}}.
This method's return type was changed in newer version of HBase. Even though
HBASE-9142 introduces the original method back in 0.95.2, it's deprecated.
Do you think it makes more sense to use {{getFamilyCellMap()}} here instead?
> HBase tests fail against Hadoop 2
> -
>
> Key: HIVE-4388
> URL: https://issues.apache.org/jira/browse/HIVE-4388
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Gunther Hagleitner
>Assignee: Brock Noland
> Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch,
> HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388-wip.txt
>
>
> Currently we're building by default against 0.92. When you run against hadoop
> 2 (-Dhadoop.mr.rev=23) builds fail because of: HBASE-5963.
> HIVE-3861 upgrades the version of hbase used. This will get you past the
> problem in HBASE-5963 (which was fixed in 0.94.1) but fails with: HBASE-6396.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737008#comment-13737008
]
Mark Grover commented on HIVE-5046:
---
Thanks Brock!
> Hcatalog's bin/hcat script doesn't respect HIVE_HOME
>
>
> Key: HIVE-5046
> URL: https://issues.apache.org/jira/browse/HIVE-5046
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Fix For: 0.12.0
>
> Attachments: HIVE-5046.1.patch
>
>
> https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
> The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
> HIVE_HOME is currently null).
> {code}
> if [ -n ${HIVE_HOME} ]; then
> {code}
> However, {{-n}} checks if the variable is _not_ null. So, the above code ends
> up setting HIVE_HOME to the default value if it is actually set already,
> overriding the set value. This condition needs to be negated.
> Moreover, {{-n}} checks requires the string being tested to be enclosed in
> quotes.
> Reference:
> http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735566#comment-13735566
]
Mark Grover commented on HIVE-5046:
---
Eugene, thanks for posting. I don't think so. The intent of the snippet you
mentioned is to do something with the variable ({{HIVE_IN_PATH}}) if it is
defined. That something is to make use to {{HIVE_IN_PATH}} to popualate
{{HIVE_DIR}}.
On the other hand, the intent of the snippet 12 lines later (that this JIRA
addresses) is to do something if the variable is not defined. That something is
to set HIVE_HOME to some default value. This snippet however, checks if the
variable is not null instead of checking if the variable is null.
> Hcatalog's bin/hcat script doesn't respect HIVE_HOME
>
>
> Key: HIVE-5046
> URL: https://issues.apache.org/jira/browse/HIVE-5046
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Attachments: HIVE-5046.1.patch
>
>
> https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
> The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
> HIVE_HOME is currently null).
> {code}
> if [ -n ${HIVE_HOME} ]; then
> {code}
> However, {{-n}} checks if the variable is _not_ null. So, the above code ends
> up setting HIVE_HOME to the default value if it is actually set already,
> overriding the set value. This condition needs to be negated.
> Moreover, {{-n}} checks requires the string being tested to be enclosed in
> quotes.
> Reference:
> http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-5046:
--
Attachment: HIVE-5046.1.patch
> Hcatalog's bin/hcat script doesn't respect HIVE_HOME
>
>
> Key: HIVE-5046
> URL: https://issues.apache.org/jira/browse/HIVE-5046
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Attachments: HIVE-5046.1.patch
>
>
> https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
> The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
> HIVE_HOME is currently null).
> {code}
> if [ -n ${HIVE_HOME} ]; then
> {code}
> However, {{-n}} checks if the variable is _not_ null. So, the above code ends
> up setting HIVE_HOME to the default value if it is actually set already,
> overriding the set value. This condition needs to be negated.
> Moreover, {{-n}} checks requires the string being tested to be enclosed in
> quotes.
> Reference:
> http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-5046:
--
Status: Patch Available (was: Open)
> Hcatalog's bin/hcat script doesn't respect HIVE_HOME
>
>
> Key: HIVE-5046
> URL: https://issues.apache.org/jira/browse/HIVE-5046
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Attachments: HIVE-5046.1.patch
>
>
> https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
> The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
> HIVE_HOME is currently null).
> {code}
> if [ -n ${HIVE_HOME} ]; then
> {code}
> However, {{-n}} checks if the variable is _not_ null. So, the above code ends
> up setting HIVE_HOME to the default value if it is actually set already,
> overriding the set value. This condition needs to be negated.
> Moreover, {{-n}} checks requires the string being tested to be enclosed in
> quotes.
> Reference:
> http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-5046:
--
Description:
https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
HIVE_HOME is currently null).
{code}
if [ -n ${HIVE_HOME} ]; then
{code}
However, {{-n}} checks if the variable is _not_ null. So, the above code ends
up setting HIVE_HOME to the default value if it is actually set already,
overriding the set value. This condition needs to be negated.
Moreover, {{-n}} checks requires the string being tested to be enclosed in
quotes.
Reference:
http://tldp.org/LDP/abs/html/comparison-ops.html
was:
https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
HIVE_HOME is currently null).
{code}
if [ -n ${HIVE_HOME} ]; then
{code}
However, {{-n}} checks if the variable is _not_ null. So, the condition is
checking for the boolean inverse of what it's supposed to be checking for.
Moreover, {{-n}} checks requires the string being tested to be enclosed in
quotes.
Reference:
http://tldp.org/LDP/abs/html/comparison-ops.html
> Hcatalog's bin/hcat script doesn't respect HIVE_HOME
>
>
> Key: HIVE-5046
> URL: https://issues.apache.org/jira/browse/HIVE-5046
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.11.0
>Reporter: Mark Grover
>Assignee: Mark Grover
>
> https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
> The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
> HIVE_HOME is currently null).
> {code}
> if [ -n ${HIVE_HOME} ]; then
> {code}
> However, {{-n}} checks if the variable is _not_ null. So, the above code ends
> up setting HIVE_HOME to the default value if it is actually set already,
> overriding the set value. This condition needs to be negated.
> Moreover, {{-n}} checks requires the string being tested to be enclosed in
> quotes.
> Reference:
> http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
Mark Grover created HIVE-5046:
-
Summary: Hcatalog's bin/hcat script doesn't respect HIVE_HOME
Key: HIVE-5046
URL: https://issues.apache.org/jira/browse/HIVE-5046
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Reporter: Mark Grover
Assignee: Mark Grover
https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
HIVE_HOME is currently null).
{code}
if [ -n ${HIVE_HOME} ]; then
{code}
However, {{-n}} checks if the variable is _not_ null. So, the condition is
checking for the boolean inverse of what it's supposed to be checking for.
Moreover, {{-n}} checks requires the string being tested to be enclosed in
quotes.
Reference:
http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4403) Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters
[
https://issues.apache.org/jira/browse/HIVE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13674027#comment-13674027
]
Mark Grover commented on HIVE-4403:
---
Chu, that's fine. Nothing required from your side.
> Running Hive queries on Yarn (MR2) gives warnings related to overriding final
> parameters
>
>
> Key: HIVE-4403
> URL: https://issues.apache.org/jira/browse/HIVE-4403
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0, 0.11.0
>Reporter: Mark Grover
>Assignee: Chu Tong
> Fix For: 0.12.0
>
> Attachments: HIVE-4403.patch, HIVE-4403.patch
>
>
> While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
> related to overriding final parameters in job.conf. This was on a pseudo
> distributed cluster. FWIW, I didn't see this happen on a fully-distributed
> cluster. Perhaps, Hive's job.conf is overriding some final parameters it
> shouldn't.
> Here is what the warnings looked like:
> {code}
> 2013-04-19 14:20:32,304 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 2013-04-19 14:20:32,367 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> {code}
> To reproduce, run a query like:
> {code}
> CREATE TABLE u_data (
> userid INT,
> movieid INT,
> rating INT,
> unixtime STRING)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> {code}
> Load some data into u_data, here is some sample data:
> https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
> Run a simple query on that data (on YARN/MR2)
> {code}
> INSERT OVERWRITE DIRECTORY '/tmp/count'
> SELECT COUNT(1) FROM u_data
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4403) Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters
[
https://issues.apache.org/jira/browse/HIVE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13672703#comment-13672703
]
Mark Grover commented on HIVE-4403:
---
Ashutosh +1'ed it. So, I guess there is no need for Phabricator now:-)
Thanks Chu and Ashutosh!
> Running Hive queries on Yarn (MR2) gives warnings related to overriding final
> parameters
>
>
> Key: HIVE-4403
> URL: https://issues.apache.org/jira/browse/HIVE-4403
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>Assignee: Chu Tong
> Attachments: HIVE-4403.patch, HIVE-4403.patch
>
>
> While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
> related to overriding final parameters in job.conf. This was on a pseudo
> distributed cluster. FWIW, I didn't see this happen on a fully-distributed
> cluster. Perhaps, Hive's job.conf is overriding some final parameters it
> shouldn't.
> Here is what the warnings looked like:
> {code}
> 2013-04-19 14:20:32,304 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 2013-04-19 14:20:32,367 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> {code}
> To reproduce, run a query like:
> {code}
> CREATE TABLE u_data (
> userid INT,
> movieid INT,
> rating INT,
> unixtime STRING)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> {code}
> Load some data into u_data, here is some sample data:
> https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
> Run a simple query on that data (on YARN/MR2)
> {code}
> INSERT OVERWRITE DIRECTORY '/tmp/count'
> SELECT COUNT(1) FROM u_data
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4403) Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters
[
https://issues.apache.org/jira/browse/HIVE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669521#comment-13669521
]
Mark Grover commented on HIVE-4403:
---
Chu, sorry about the delay. I can test patched Hive to see if this resolves my
original problem, please give me a few days.
In the meanwhile, even though this is a small patch, do you mind uploading it
on Reviewboard or Phabricator? Thanks for contributing!
> Running Hive queries on Yarn (MR2) gives warnings related to overriding final
> parameters
>
>
> Key: HIVE-4403
> URL: https://issues.apache.org/jira/browse/HIVE-4403
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mark Grover
> Attachments: HIVE-4403.patch
>
>
> While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
> related to overriding final parameters in job.conf. This was on a pseudo
> distributed cluster. FWIW, I didn't see this happen on a fully-distributed
> cluster. Perhaps, Hive's job.conf is overriding some final parameters it
> shouldn't.
> Here is what the warnings looked like:
> {code}
> 2013-04-19 14:20:32,304 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 2013-04-19 14:20:32,367 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> {code}
> To reproduce, run a query like:
> {code}
> CREATE TABLE u_data (
> userid INT,
> movieid INT,
> rating INT,
> unixtime STRING)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> {code}
> Load some data into u_data, here is some sample data:
> https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
> Run a simple query on that data (on YARN/MR2)
> {code}
> INSERT OVERWRITE DIRECTORY '/tmp/count'
> SELECT COUNT(1) FROM u_data
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4070) Like operator in Hive is case sensitive while in MySQL (and most likely other DBs) it's case insensitive
[ https://issues.apache.org/jira/browse/HIVE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666785#comment-13666785 ] Mark Grover commented on HIVE-4070: --- I agree with Edward that having a property like that is risky. I also like John's idea of having two different like UDFs. We can only change the behavior (if at all) in a major release. I think in this case, we are just going to have to pick one kind of like and implement another one. Yeah, people coming from MySQL land may get confused but that's the best we can do is document it and ask them to use the other like. John, it would be great if you could create a JIRA for the case-insensitive like. Thanks! > Like operator in Hive is case sensitive while in MySQL (and most likely other > DBs) it's case insensitive > > > Key: HIVE-4070 > URL: https://issues.apache.org/jira/browse/HIVE-4070 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover >Priority: Trivial > > Hive's like operator seems to be case sensitive. > See > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L164 > However, MySQL's like operator is case insensitive. I don't have other DB's > (like PostgreSQL) installed and handy but I am guessing their LIKE is case > insensitive as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649880#comment-13649880 ] Mark Grover commented on HIVE-3384: --- I verified that this was committed on trunk (Thanks!). Is there anything left to be done regarding this patch? If not, can we please update the JIRA to resolved status? > HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC > specification > -- > > Key: HIVE-3384 > URL: https://issues.apache.org/jira/browse/HIVE-3384 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 0.10.0 >Reporter: Weidong Bian >Assignee: Chris Drome >Priority: Minor > Fix For: 0.11.0 > > Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, > D6873.3.patch, D6873.4.patch, D6873.5.patch, D6873.6.patch, D6873.7.patch, > HIVE-3384-0.10.patch, HIVE-3384-2012-12-02.patch, HIVE-3384-2012-12-04.patch, > HIVE-3384.2.patch, HIVE-3384-branch-0.9.patch, HIVE-3384.patch, > HIVE-JDK7-JDBC.patch > > > jdbc module couldn't be compiled with jdk7 as it adds some abstract method in > the JDBC specification > some error info: > error: HiveCallableStatement is not abstract and does not override abstract > method getObject(String,Class) in CallableStatement > . > . > . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4403) Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters
[
https://issues.apache.org/jira/browse/HIVE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-4403:
--
Description:
While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
related to overriding final parameters in job.conf. This was on a pseudo
distributed cluster. FWIW, I didn't see this happen on a fully-distributed
cluster. Perhaps, Hive's job.conf is overriding some final parameters it
shouldn't.
Here is what the warnings looked like:
{code}
2013-04-19 14:20:32,304 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2032)) -
file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2013-04-19 14:20:32,367 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2032)) -
file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
{code}
To reproduce, run a query like:
{code}
CREATE TABLE u_data (
userid INT,
movieid INT,
rating INT,
unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
{code}
Load some data into u_data, here is some sample data:
https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
Run a simple query on that data (on YARN/MR2)
{code}
INSERT OVERWRITE DIRECTORY '/tmp/count'
SELECT COUNT(1) FROM u_data
{code}
was:
While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
related to overriding final parameters in job.conf. Perhaps, Hive's job.conf is
overriding some final parameters it shouldn't.
Here is what the warnings looked like:
{code}
2013-04-19 14:20:32,304 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2032)) -
file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2013-04-19 14:20:32,367 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2032)) -
file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
{code}
To reproduce, run a query like:
{code}
CREATE TABLE u_data (
userid INT,
movieid INT,
rating INT,
unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
{code}
Load some data into u_data, here is some sample data:
https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
Run a simple query on that data (on YARN/MR2)
{code}
INSERT OVERWRITE DIRECTORY '/tmp/count'
SELECT COUNT(1) FROM u_data
{code}
> Running Hive queries on Yarn (MR2) gives warnings related to overriding final
> parameters
>
>
> Key: HIVE-4403
> URL: https://issues.apache.org/jira/browse/HIVE-4403
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>
> While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
> related to overriding final parameters in job.conf. This was on a pseudo
> distributed cluster. FWIW, I didn't see this happen on a fully-distributed
> cluster. Perhaps, Hive's job.conf is overriding some final parameters it
> shouldn't.
> Here is what the warnings looked like:
> {code}
> 2013-04-19 14:20:32,304 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 2013-04-19 14:20:32,367 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> {code}
> To reproduce, run a query like:
> {code}
> CREATE TABLE u_data (
> userid INT,
> movieid INT,
> rating INT,
> unixtime STRING)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> {code}
> Load some data into u_data, here is some sample data:
> https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
> Run a simple query on that data (on YARN/MR2)
> {code}
> INSERT OVERWRITE DIRECTORY '/tmp/count'
> SELECT COUNT(1) FROM u_data
> {code}
--
This message is automatically gen
[jira] [Updated] (HIVE-4403) Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters
[
https://issues.apache.org/jira/browse/HIVE-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-4403:
--
Affects Version/s: 0.10.0
> Running Hive queries on Yarn (MR2) gives warnings related to overriding final
> parameters
>
>
> Key: HIVE-4403
> URL: https://issues.apache.org/jira/browse/HIVE-4403
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>
> While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
> related to overriding final parameters in job.conf. Perhaps, Hive's job.conf
> is overriding some final parameters it shouldn't.
> Here is what the warnings looked like:
> {code}
> 2013-04-19 14:20:32,304 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval; Ignoring.
> 2013-04-19 14:20:32,367 WARN [main] conf.Configuration
> (Configuration.java:loadProperty(2032)) -
> file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
> attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts; Ignoring.
> {code}
> To reproduce, run a query like:
> {code}
> CREATE TABLE u_data (
> userid INT,
> movieid INT,
> rating INT,
> unixtime STRING)
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
> {code}
> Load some data into u_data, here is some sample data:
> https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
> Run a simple query on that data (on YARN/MR2)
> {code}
> INSERT OVERWRITE DIRECTORY '/tmp/count'
> SELECT COUNT(1) FROM u_data
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4403) Running Hive queries on Yarn (MR2) gives warnings related to overriding final parameters
Mark Grover created HIVE-4403:
-
Summary: Running Hive queries on Yarn (MR2) gives warnings related
to overriding final parameters
Key: HIVE-4403
URL: https://issues.apache.org/jira/browse/HIVE-4403
Project: Hive
Issue Type: Bug
Reporter: Mark Grover
While working on BIGTOP-885, I saw that Hive was giving a bunch of warnings
related to overriding final parameters in job.conf. Perhaps, Hive's job.conf is
overriding some final parameters it shouldn't.
Here is what the warnings looked like:
{code}
2013-04-19 14:20:32,304 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2032)) -
file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2013-04-19 14:20:32,367 WARN [main] conf.Configuration
(Configuration.java:loadProperty(2032)) -
file:/tmp/root/hive_2013-04-19_14-20-30_159_5701876916688815815/-local-10002/jobconf.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
{code}
To reproduce, run a query like:
{code}
CREATE TABLE u_data (
userid INT,
movieid INT,
rating INT,
unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
{code}
Load some data into u_data, here is some sample data:
https://github.com/apache/bigtop/blob/master/bigtop-tests/test-artifacts/hive/src/main/resources/seed_data_files/ml-data/u.data
Run a simple query on that data (on YARN/MR2)
{code}
INSERT OVERWRITE DIRECTORY '/tmp/count'
SELECT COUNT(1) FROM u_data
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4362) Allow Hive unit tests to run against fully-distributed cluster
Mark Grover created HIVE-4362: - Summary: Allow Hive unit tests to run against fully-distributed cluster Key: HIVE-4362 URL: https://issues.apache.org/jira/browse/HIVE-4362 Project: Hive Issue Type: Improvement Components: Testing Infrastructure Affects Versions: 0.10.0 Reporter: Mark Grover Assignee: Mark Grover Fix For: 0.11.0 It seems like Hive unit tests can run in (Hadoop) local mode or miniMR mode. It would be nice (especially for projects like Apache Bigtop) to be able to run Hive tests in fully distributed mode. This JIRA tracks the introduction of such functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4213) List bucketing error too restrictive
[
https://issues.apache.org/jira/browse/HIVE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621158#comment-13621158
]
Mark Grover commented on HIVE-4213:
---
Hi [~gangtimliu], really sorry for dropping the ball on this. I do see what you
mean by the legal cases but is it legal to do this:
{code}
set hive.mapred.supports.subdirectories=false;
set mapred.input.dir.recursive=true;
set hive.optimize.listbucketing=false;
{code}
> List bucketing error too restrictive
>
>
> Key: HIVE-4213
> URL: https://issues.apache.org/jira/browse/HIVE-4213
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>Assignee: Gang Tim Liu
> Fix For: 0.11.0
>
>
> With the introduction of List bucketing, we introduced a config validation
> step where we say:
> {code}
> SUPPORT_DIR_MUST_TRUE_FOR_LIST_BUCKETING(
> 10199,
> "hive.mapred.supports.subdirectories must be true"
> + " if any one of following is true:
> hive.internal.ddl.list.bucketing.enable,"
> + " hive.optimize.listbucketing and mapred.input.dir.recursive"),
> {code}
> This seems overly restrictive to because there are use cases where people may
> want to use {{mapred.input.dir.recursive}} to {{true}} even when they don't
> care about list bucketing.
> Is that not true?
> For example, here is the unit test code for {{clientpositive/recursive_dir.q}}
> {code}
> CREATE TABLE fact_daily(x int) PARTITIONED BY (ds STRING);
> CREATE TABLE fact_tz(x int) PARTITIONED BY (ds STRING, hr STRING)
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz';
> INSERT OVERWRITE TABLE fact_tz PARTITION (ds='1', hr='1')
> SELECT key+11 FROM src WHERE key=484;
> ALTER TABLE fact_daily SET TBLPROPERTIES('EXTERNAL'='TRUE');
> ALTER TABLE fact_daily ADD PARTITION (ds='1')
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz/ds=1';
> set hive.mapred.supports.subdirectories=true;
> set mapred.input.dir.recursive=true;
> set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
> SELECT * FROM fact_daily WHERE ds='1';
> SELECT count(1) FROM fact_daily WHERE ds='1';
> {code}
> The unit test doesn't seem to be concerned about list bucketing but wants to
> set {{mapred.input.dir.recursive}} to {{true}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4174) Round UDF converts BigInts to double
[
https://issues.apache.org/jira/browse/HIVE-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621137#comment-13621137
]
Mark Grover commented on HIVE-4174:
---
It should be. [~namitjain] can you please add Chen Chun (chenchun dot feed at
gmail dot com) to the project contributors list and assign this JIRA to him?
Thanks!
> Round UDF converts BigInts to double
>
>
> Key: HIVE-4174
> URL: https://issues.apache.org/jira/browse/HIVE-4174
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Fix For: 0.11.0
>
> Attachments: hive.4174.1.patch-nohcat, HIVE-4174.1.patch.txt,
> HIVE-4174.D9687.1.patch
>
>
> Chen Chun pointed out on the hive-user mailing list that round() in Hive 0.10
> returns
> {code}
> select round(cast(1234560 as BIGINT)), round(cast(12345670 as BIGINT)) from
> test limit 1;
> //hive 0.10
> 1234560.0 1.234567E7
> {code}
> This is not consistent with
> MySQL(http://dev.mysql.com/doc/refman/5.1/en/mathematical-functions.html#function_round)
> which quotes
> {code}
> The return type is the same type as that of the first argument (assuming that
> it is integer, double, or decimal). This means that for an integer argument,
> the result is an integer (no decimal places)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4174) Round UDF converts BigInts to double
[
https://issues.apache.org/jira/browse/HIVE-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621069#comment-13621069
]
Mark Grover commented on HIVE-4174:
---
Indeed, thanks Chen Chun. And, congratulations on your first Hive commit!
> Round UDF converts BigInts to double
>
>
> Key: HIVE-4174
> URL: https://issues.apache.org/jira/browse/HIVE-4174
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Fix For: 0.11.0
>
> Attachments: hive.4174.1.patch-nohcat, HIVE-4174.1.patch.txt,
> HIVE-4174.D9687.1.patch
>
>
> Chen Chun pointed out on the hive-user mailing list that round() in Hive 0.10
> returns
> {code}
> select round(cast(1234560 as BIGINT)), round(cast(12345670 as BIGINT)) from
> test limit 1;
> //hive 0.10
> 1234560.0 1.234567E7
> {code}
> This is not consistent with
> MySQL(http://dev.mysql.com/doc/refman/5.1/en/mathematical-functions.html#function_round)
> which quotes
> {code}
> The return type is the same type as that of the first argument (assuming that
> it is integer, double, or decimal). This means that for an integer argument,
> the result is an integer (no decimal places)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619518#comment-13619518 ] Mark Grover commented on HIVE-3951: --- Thanks, Ashutosh! > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch, HIVE-3951.2.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618821#comment-13618821 ] Mark Grover commented on HIVE-3951: --- RB updated > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch, HIVE-3951.2.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3951: -- Attachment: HIVE-3951.2.patch Thanks for the comments, Ashutosh. Uploaded a new patch. Will update Review Board momentarily. > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch, HIVE-3951.2.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3451) map-reduce jobs does not work for a partition containing sub-directories
[
https://issues.apache.org/jira/browse/HIVE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover reassigned HIVE-3451:
-
Assignee: Gang Tim Liu (was: Mark Grover)
> map-reduce jobs does not work for a partition containing sub-directories
>
>
> Key: HIVE-3451
> URL: https://issues.apache.org/jira/browse/HIVE-3451
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Namit Jain
>Assignee: Gang Tim Liu
> Fix For: 0.10.0
>
> Attachments: HIVE-3451.patch
>
>
> Consider the following test:
> -- The test verifies that sub-directories are supported for versions of hadoop
> -- where MAPREDUCE-1501 is fixed. So, enable this test only for hadoop 23.
> -- INCLUDE_HADOOP_MAJOR_VERSIONS(0.23)
> CREATE TABLE fact_daily(x int) PARTITIONED BY (ds STRING);
> CREATE TABLE fact_tz(x int) PARTITIONED BY (ds STRING, hr STRING)
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz';
> INSERT OVERWRITE TABLE fact_tz PARTITION (ds='1', hr='1')
> SELECT key+11 FROM src WHERE key=484;
> ALTER TABLE fact_daily SET TBLPROPERTIES('EXTERNAL'='TRUE');
> ALTER TABLE fact_daily ADD PARTITION (ds='1')
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz/ds=1';
> set mapred.input.dir.recursive=true;
> SELECT * FROM fact_daily WHERE ds='1';
> SELECT count(1) FROM fact_daily WHERE ds='1';
> Say, the above file was named: recursive_dir.q
> and we ran the test for hadoop 23:
> by executing:
> ant test -Dhadoop.mr.rev=23 -Dtest.print.classpath=true
> -Dhadoop.version=2.0.0-alpha -Dhadoop.security.version=2.0.0-alpha
> -Dtestcase=TestCliDriver -Dqfile=recursive_dir.q
> The select * from the table works fine, but the last command does not work
> since it requires a map-reduce job.
> This will prevent other features which are creating sub-directories to add
> any tests which requires a map-reduce job. The work-around is to issue
> queries which do not require map-reduce jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3451) map-reduce jobs does not work for a partition containing sub-directories
[
https://issues.apache.org/jira/browse/HIVE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover reassigned HIVE-3451:
-
Assignee: Mark Grover (was: Gang Tim Liu)
> map-reduce jobs does not work for a partition containing sub-directories
>
>
> Key: HIVE-3451
> URL: https://issues.apache.org/jira/browse/HIVE-3451
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Namit Jain
>Assignee: Mark Grover
> Fix For: 0.10.0
>
> Attachments: HIVE-3451.patch
>
>
> Consider the following test:
> -- The test verifies that sub-directories are supported for versions of hadoop
> -- where MAPREDUCE-1501 is fixed. So, enable this test only for hadoop 23.
> -- INCLUDE_HADOOP_MAJOR_VERSIONS(0.23)
> CREATE TABLE fact_daily(x int) PARTITIONED BY (ds STRING);
> CREATE TABLE fact_tz(x int) PARTITIONED BY (ds STRING, hr STRING)
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz';
> INSERT OVERWRITE TABLE fact_tz PARTITION (ds='1', hr='1')
> SELECT key+11 FROM src WHERE key=484;
> ALTER TABLE fact_daily SET TBLPROPERTIES('EXTERNAL'='TRUE');
> ALTER TABLE fact_daily ADD PARTITION (ds='1')
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz/ds=1';
> set mapred.input.dir.recursive=true;
> SELECT * FROM fact_daily WHERE ds='1';
> SELECT count(1) FROM fact_daily WHERE ds='1';
> Say, the above file was named: recursive_dir.q
> and we ran the test for hadoop 23:
> by executing:
> ant test -Dhadoop.mr.rev=23 -Dtest.print.classpath=true
> -Dhadoop.version=2.0.0-alpha -Dhadoop.security.version=2.0.0-alpha
> -Dtestcase=TestCliDriver -Dqfile=recursive_dir.q
> The select * from the table works fine, but the last command does not work
> since it requires a map-reduce job.
> This will prevent other features which are creating sub-directories to add
> any tests which requires a map-reduce job. The work-around is to issue
> queries which do not require map-reduce jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4070) Like operator in Hive is case sensitive while in MySQL (and most likely other DBs) it's case insensitive
[ https://issues.apache.org/jira/browse/HIVE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609278#comment-13609278 ] Mark Grover commented on HIVE-4070: --- Thanks Gwen. In that case, I am ok with documenting it and resolving this JIRA as won't fix. Is that ok with you as well, [~mackrorysd]? > Like operator in Hive is case sensitive while in MySQL (and most likely other > DBs) it's case insensitive > > > Key: HIVE-4070 > URL: https://issues.apache.org/jira/browse/HIVE-4070 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover >Priority: Trivial > Fix For: 0.11.0 > > > Hive's like operator seems to be case sensitive. > See > https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L164 > However, MySQL's like operator is case insensitive. I don't have other DB's > (like PostgreSQL) installed and handy but I am guessing their LIKE is case > insensitive as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4213) List bucketing error too restrictive
[
https://issues.apache.org/jira/browse/HIVE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609262#comment-13609262
]
Mark Grover commented on HIVE-4213:
---
[~gangtimliu] I would love to hear your thoughts on this! Thanks in advance!
> List bucketing error too restrictive
>
>
> Key: HIVE-4213
> URL: https://issues.apache.org/jira/browse/HIVE-4213
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Mark Grover
> Fix For: 0.11.0
>
>
> With the introduction of List bucketing, we introduced a config validation
> step where we say:
> {code}
> SUPPORT_DIR_MUST_TRUE_FOR_LIST_BUCKETING(
> 10199,
> "hive.mapred.supports.subdirectories must be true"
> + " if any one of following is true:
> hive.internal.ddl.list.bucketing.enable,"
> + " hive.optimize.listbucketing and mapred.input.dir.recursive"),
> {code}
> This seems overly restrictive to because there are use cases where people may
> want to use {{mapred.input.dir.recursive}} to {{true}} even when they don't
> care about list bucketing.
> Is that not true?
> For example, here is the unit test code for {{clientpositive/recursive_dir.q}}
> {code}
> CREATE TABLE fact_daily(x int) PARTITIONED BY (ds STRING);
> CREATE TABLE fact_tz(x int) PARTITIONED BY (ds STRING, hr STRING)
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz';
> INSERT OVERWRITE TABLE fact_tz PARTITION (ds='1', hr='1')
> SELECT key+11 FROM src WHERE key=484;
> ALTER TABLE fact_daily SET TBLPROPERTIES('EXTERNAL'='TRUE');
> ALTER TABLE fact_daily ADD PARTITION (ds='1')
> LOCATION 'pfile:${system:test.tmp.dir}/fact_tz/ds=1';
> set hive.mapred.supports.subdirectories=true;
> set mapred.input.dir.recursive=true;
> set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
> SELECT * FROM fact_daily WHERE ds='1';
> SELECT count(1) FROM fact_daily WHERE ds='1';
> {code}
> The unit test doesn't seem to be concerned about list bucketing but wants to
> set {{mapred.input.dir.recursive}} to {{true}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4213) List bucketing error too restrictive
Mark Grover created HIVE-4213:
-
Summary: List bucketing error too restrictive
Key: HIVE-4213
URL: https://issues.apache.org/jira/browse/HIVE-4213
Project: Hive
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Mark Grover
Fix For: 0.11.0
With the introduction of List bucketing, we introduced a config validation step
where we say:
{code}
SUPPORT_DIR_MUST_TRUE_FOR_LIST_BUCKETING(
10199,
"hive.mapred.supports.subdirectories must be true"
+ " if any one of following is true:
hive.internal.ddl.list.bucketing.enable,"
+ " hive.optimize.listbucketing and mapred.input.dir.recursive"),
{code}
This seems overly restrictive to because there are use cases where people may
want to use {{mapred.input.dir.recursive}} to {{true}} even when they don't
care about list bucketing.
Is that not true?
For example, here is the unit test code for {{clientpositive/recursive_dir.q}}
{code}
CREATE TABLE fact_daily(x int) PARTITIONED BY (ds STRING);
CREATE TABLE fact_tz(x int) PARTITIONED BY (ds STRING, hr STRING)
LOCATION 'pfile:${system:test.tmp.dir}/fact_tz';
INSERT OVERWRITE TABLE fact_tz PARTITION (ds='1', hr='1')
SELECT key+11 FROM src WHERE key=484;
ALTER TABLE fact_daily SET TBLPROPERTIES('EXTERNAL'='TRUE');
ALTER TABLE fact_daily ADD PARTITION (ds='1')
LOCATION 'pfile:${system:test.tmp.dir}/fact_tz/ds=1';
set hive.mapred.supports.subdirectories=true;
set mapred.input.dir.recursive=true;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
SELECT * FROM fact_daily WHERE ds='1';
SELECT count(1) FROM fact_daily WHERE ds='1';
{code}
The unit test doesn't seem to be concerned about list bucketing but wants to
set {{mapred.input.dir.recursive}} to {{true}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4208) Clientpositive test parenthesis_star_by is non-deteministic
[
https://issues.apache.org/jira/browse/HIVE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-4208:
--
Attachment: HIVE-4208.1.patch
Uploading patch, adding {{ORDER BY}} to make the order of rows in the result
deterministic.
> Clientpositive test parenthesis_star_by is non-deteministic
> ---
>
> Key: HIVE-4208
> URL: https://issues.apache.org/jira/browse/HIVE-4208
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Fix For: 0.11.0
>
> Attachments: HIVE-4208.1.patch
>
>
> parenthesis_star_by is testing {{DISTRIBUTE BY}}; however, the order of rows
> returned by {{DISTRIBUTE BY}} is not deterministic and results in failures
> depending on Hadoop version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4208) Clientpositive test parenthesis_star_by is non-deteministic
[
https://issues.apache.org/jira/browse/HIVE-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-4208:
--
Status: Patch Available (was: Open)
[~namitjain] since you were the original author of this test, would you mind
reviewing it please?
> Clientpositive test parenthesis_star_by is non-deteministic
> ---
>
> Key: HIVE-4208
> URL: https://issues.apache.org/jira/browse/HIVE-4208
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Affects Versions: 0.10.0
>Reporter: Mark Grover
>Assignee: Mark Grover
> Fix For: 0.11.0
>
> Attachments: HIVE-4208.1.patch
>
>
> parenthesis_star_by is testing {{DISTRIBUTE BY}}; however, the order of rows
> returned by {{DISTRIBUTE BY}} is not deterministic and results in failures
> depending on Hadoop version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4208) Clientpositive test parenthesis_star_by is non-deteministic
Mark Grover created HIVE-4208:
-
Summary: Clientpositive test parenthesis_star_by is
non-deteministic
Key: HIVE-4208
URL: https://issues.apache.org/jira/browse/HIVE-4208
Project: Hive
Issue Type: Bug
Components: Tests
Affects Versions: 0.10.0
Reporter: Mark Grover
Assignee: Mark Grover
Fix For: 0.11.0
parenthesis_star_by is testing {{DISTRIBUTE BY}}; however, the order of rows
returned by {{DISTRIBUTE BY}} is not deterministic and results in failures
depending on Hadoop version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4174) Round UDF converts BigInts to double
Mark Grover created HIVE-4174:
-
Summary: Round UDF converts BigInts to double
Key: HIVE-4174
URL: https://issues.apache.org/jira/browse/HIVE-4174
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.10.0
Reporter: Mark Grover
Assignee: Mark Grover
Fix For: 0.11.0
Chen Chun pointed out on the hive-user mailing list that round() in Hive 0.10
returns
{code}
select round(cast(1234560 as BIGINT)), round(cast(12345670 as BIGINT)) from
test limit 1;
//hive 0.10
1234560.0 1.234567E7
{code}
This is not consistent with
MySQL(http://dev.mysql.com/doc/refman/5.1/en/mathematical-functions.html#function_round)
which quotes
{code}
The return type is the same type as that of the first argument (assuming that
it is integer, double, or decimal). This means that for an integer argument,
the result is an integer (no decimal places)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599022#comment-13599022
]
Mark Grover commented on HIVE-3963:
---
Thanks Maxime. Can you also please post the patch on reviewboard (or
Phabricator) as well?
> Allow Hive to connect to RDBMS
> --
>
> Key: HIVE-3963
> URL: https://issues.apache.org/jira/browse/HIVE-3963
> Project: Hive
> Issue Type: New Feature
> Components: Import/Export, JDBC, SQL, StorageHandler
>Affects Versions: 0.10.0, 0.9.1, 0.11.0
>Reporter: Maxime LANCIAUX
> Fix For: 0.10.1
>
> Attachments: patchfile
>
>
> I am thinking about something like :
> SELECT jdbcload('driver','url','user','password','sql') FROM dual;
> There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
> JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3963) Allow Hive to connect to RDBMS
[
https://issues.apache.org/jira/browse/HIVE-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598050#comment-13598050
]
Mark Grover commented on HIVE-3963:
---
Maxime, can you upload a patch and post it for review.
Also, out of curiosity, when would a user use this instead of using something
like Apache Sqoop.
> Allow Hive to connect to RDBMS
> --
>
> Key: HIVE-3963
> URL: https://issues.apache.org/jira/browse/HIVE-3963
> Project: Hive
> Issue Type: New Feature
> Components: Import/Export, JDBC, SQL, StorageHandler
>Affects Versions: 0.9.0, 0.10.0, 0.9.1, 0.11.0
>Reporter: Maxime LANCIAUX
>
> I am thinking about something like :
> SELECT jdbcload('driver','url','user','password','sql') FROM dual;
> There is already a JIRA https://issues.apache.org/jira/browse/HIVE-1555 for
> JDBCStorageHandler
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3850) hour() function returns 12 hour clock value when using timestamp datatype
[ https://issues.apache.org/jira/browse/HIVE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13598049#comment-13598049 ] Mark Grover commented on HIVE-3850: --- +1 (non-committer) > hour() function returns 12 hour clock value when using timestamp datatype > - > > Key: HIVE-3850 > URL: https://issues.apache.org/jira/browse/HIVE-3850 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.9.0, 0.10.0 >Reporter: Pieterjan Vriends > Fix For: 0.11.0 > > Attachments: hive-3850_1.patch, HIVE-3850.patch.txt > > > Apparently UDFHour.java does have two evaluate() functions. One that does > accept a Text object as parameter and one that does use a TimeStampWritable > object as parameter. The first function does return the value of > Calendar.HOUR_OF_DAY and the second one of Calendar.HOUR. In the > documentation I couldn't find any information on the overload of the > evaluation function. I did spent quite some time finding out why my statement > didn't return a 24 hour clock value. > Shouldn't both functions return the same? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4144) Add "select database()" command to show the current database
Mark Grover created HIVE-4144:
-
Summary: Add "select database()" command to show the current
database
Key: HIVE-4144
URL: https://issues.apache.org/jira/browse/HIVE-4144
Project: Hive
Issue Type: Bug
Components: SQL
Reporter: Mark Grover
A recent hive-user mailing list conversation asked about having a command to
show the current database.
http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0i+CRY69m3id=DxthmUCWLf0NxpKMCtROb=uauh2va...@mail.gmail.com%3E
MySQL seems to have a command to do so:
{code}
select database();
{code}
http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_database
We should look into having something similar in Hive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4053) Add support for phonetic algorithms in Hive
[ https://issues.apache.org/jira/browse/HIVE-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591449#comment-13591449 ] Mark Grover commented on HIVE-4053: --- Krishna, thanks for doing this. I don't have a whole lot of insight into these particular algorithms but do they always take the same parameters? What's the possibility of a new phonetic algorithm using a different set or number of parameters? If these functions always take same parameters, it may make sense to do (2). However, if not, (1) would be a good idea. Of course, you can still refactor the code and share amongst all different UDFs even when they are separate. To post a review on reviewboard, go to reviews.apache.org. Generate a diff file of your changes on top of hive trunk (using svn diff or git diff) and upload that diff (use "hive" repository when using svn diff output and hive-git repository when using git diff output). Please let me know if you have any further questions. > Add support for phonetic algorithms in Hive > --- > > Key: HIVE-4053 > URL: https://issues.apache.org/jira/browse/HIVE-4053 > Project: Hive > Issue Type: New Feature > Components: UDF >Affects Versions: 0.10.0 >Reporter: Krishna > Labels: patch > Fix For: 0.10.0 > > Attachments: FunctionRegistry.java, GenericUDFRefinedSoundex.java, > HIVE-4053.1.patch.txt > > > Following phonetic algorithms should be considered, which are very useful in > search: > Soundex: http://en.wikipedia.org/wiki/Soundex > Refined Soundex: Refer to the comment on 22/Feb/13 23:51 > Daitch–Mokotoff Soundex: > http://en.wikipedia.org/wiki/Daitch%E2%80%93Mokotoff_Soundex > Metaphone and Double Metaphone: http://en.wikipedia.org/wiki/Metaphone > New York State Identification and Intelligence System (NYSIIS): > http://en.wikipedia.org/wiki/New_York_State_Identification_and_Intelligence_System > Caverphone: http://en.wikipedia.org/wiki/Caverphone -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4100) Improve regex_replace UDF to allow non-ascii characters
Mark Grover created HIVE-4100: - Summary: Improve regex_replace UDF to allow non-ascii characters Key: HIVE-4100 URL: https://issues.apache.org/jira/browse/HIVE-4100 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.10.0 Reporter: Mark Grover Assignee: Mark Grover Fix For: 0.11.0 There have a been a few email threads on the user mailing list regarding regex_replace UDF not supporting non-ASCII characters. We should validate that and improve the UDF to allow it. Translate UDF will be a good reference since it does that by using code points instead of characters -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4070) Like operator in Hive is case sensitive while in MySQL (and most likely other DBs) it's case sensitive
Mark Grover created HIVE-4070: - Summary: Like operator in Hive is case sensitive while in MySQL (and most likely other DBs) it's case sensitive Key: HIVE-4070 URL: https://issues.apache.org/jira/browse/HIVE-4070 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.10.0 Reporter: Mark Grover Assignee: Mark Grover Fix For: 0.11.0 Hive's like operator seems to be case sensitive. See https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L164 However, MySQL's like operator is case insensitive. I don't have other DB's (like PostgreSQL) installed and handy but I am guessing their LIKE is case insensitive as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4053) Add support for phonetic algorithms in Hive
[ https://issues.apache.org/jira/browse/HIVE-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585226#comment-13585226 ] Mark Grover commented on HIVE-4053: --- Can you please post a patch on the JIRA and post a review on reviewboard? You should also consider adding some unit tests. If you need help with any of this, please let us know. > Add support for phonetic algorithms in Hive > --- > > Key: HIVE-4053 > URL: https://issues.apache.org/jira/browse/HIVE-4053 > Project: Hive > Issue Type: New Feature > Components: UDF >Reporter: Krishna > Attachments: FunctionRegistry.java, GenericUDFRefinedSoundex.java > > > Following phonetic algorithms should be considered, which are very useful in > search: > Soundex > Refined Soundex > Daitch–Mokotoff Soundex > Metaphone and Double Metaphone > New York State Identification and Intelligence System (NYSIIS) > Caverphone -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3885) CLI command "SHOW PARTITIONS" could be extended to provide LOCATION information
[ https://issues.apache.org/jira/browse/HIVE-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579844#comment-13579844 ] Mark Grover commented on HIVE-3885: --- Sanjay: I am replying to your question on the IRC channel. I think this will be a good starting point, https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowPartitionsDesc.java See who uses this class and how. > CLI command "SHOW PARTITIONS" could be extended to provide LOCATION > information > --- > > Key: HIVE-3885 > URL: https://issues.apache.org/jira/browse/HIVE-3885 > Project: Hive > Issue Type: New Feature >Reporter: Sanjay Subramanian >Priority: Minor > > SHOW PARTITIONS does not provide information on the HDFS location of the > data. The workaround is to query the metadata DB. The following command will > give you the HDFS file locations as stored in the metadata tables. > > echo "select t.TBL_NAME, p.PART_NAME, s.LOCATION from PARTITIONS p, SDS s, > TBLS t where t.TBL_ID=p.TBL_ID and p.SD_ID=s.SD_ID " |mysql -u > –p |grep |less > If this could be encapsulated in a CLI command SHOW LOCATIONS that displays > PARTITION_NAMELOCATION -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577324#comment-13577324 ] Mark Grover commented on HIVE-3951: --- This patch is ready for review. Would anyone be willing to please review? > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3179) HBase Handler doesn't handle NULLs properly
[
https://issues.apache.org/jira/browse/HIVE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575246#comment-13575246
]
Mark Grover commented on HIVE-3179:
---
They timed out on my pseudo-distributed laptop but that most likely is an
environment issue local to me. But looks like Lars mentioned that he had run
the tests, so that should be ok.
I will try to fix the environment, but don't wait on me.
> HBase Handler doesn't handle NULLs properly
> ---
>
> Key: HIVE-3179
> URL: https://issues.apache.org/jira/browse/HIVE-3179
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 0.9.0, 0.10.0
>Reporter: Lars Francke
>Priority: Critical
> Attachments: HIVE-3179.1.patch
>
>
> We found a quite severe issue in the HBase Handler which actually means that
> Hive potentially returns incorrect data if a column has NULL values in HBase
> (which means the cell doesn't even exist)
> In HBase Shell:
> {noformat}
> create 'hive_hbase_test', 'test'
> put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
> put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
> put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
> put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
> {noformat}
> In Hive:
> {noformat}
> DROP TABLE IF EXISTS hive_hbase_test;
> CREATE EXTERNAL TABLE hive_hbase_test (
> id int,
> c1 string,
> c2 string,
> c3 string
> )
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key#s,test:c1#s,test:c2#s,test:c3#s")
> TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");
> hive> select * from hive_hbase_test;
> OK
> 1 c1-1c2-1c3-1
> 2 c1-2NULLNULL
> hive> select c1 from hive_hbase_test;
> c1-1
> c1-2
> hive> select c1, c2 from hive_hbase_test;
> c1-1 c2-1
> c1-2 NULL
> {noformat}
> So far everything is correct but now:
> {noformat}
> hive> select c1, c2, c2 from hive_hbase_test;
> c1-1 c2-1c2-1
> c1-2 NULLc2-1
> {noformat}
> Selecting c2 twice works the first time but the second time we
> actually get the value from the previous row.
> {noformat}
> hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
> c1-1 c3-1c2-1c2-1c3-1c3-1c1-1
> c1-2 NULLNULLc2-1c3-1c3-1c1-2
> {noformat}
> We've narrowed this down to an early initialization of
> {{fieldsInited\[fieldID] = true}} in {{LazyHBaseRow#uncheckedGetField}} and
> we'll try to provide a patch which surely needs review.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3995) PostgreSQL upgrade scripts are not valid
[ https://issues.apache.org/jira/browse/HIVE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575186#comment-13575186 ] Mark Grover commented on HIVE-3995: --- Thanks, Namit! > PostgreSQL upgrade scripts are not valid > > > Key: HIVE-3995 > URL: https://issues.apache.org/jira/browse/HIVE-3995 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Jarek Jarcec Cecho > Fix For: 0.11.0 > > Attachments: postgre_update_issue.patch, postgre_update_issue.patch > > > I've noticed that scripts for upgrading metastore backed up on PostgreSQL are > not valid. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover reassigned HIVE-4003: - Assignee: Mark Grover > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575185#comment-13575185 ] Mark Grover commented on HIVE-4003: --- Ok, in that case, I can take it on from here. Thanks for your contribution, Thomas! > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam > Fix For: 0.11.0 > > Attachments: HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-4003: -- Description: Utilities.java seems to be throwing a NPE. Change contributed by Thomas Adam. Reference: https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 Affects Version/s: 0.10.0 Fix Version/s: 0.11.0 > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam > Fix For: 0.11.0 > > Attachments: HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4003) NullPointerException in ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
[ https://issues.apache.org/jira/browse/HIVE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574671#comment-13574671 ] Mark Grover commented on HIVE-4003: --- Thanks Thomas. Could you point out how one can reproduce the error? > NullPointerException in > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > - > > Key: HIVE-4003 > URL: https://issues.apache.org/jira/browse/HIVE-4003 > Project: Hive > Issue Type: Bug >Affects Versions: 0.10.0 >Reporter: Thomas Adam > Fix For: 0.11.0 > > Attachments: HIVE-4003.patch > > > Utilities.java seems to be throwing a NPE. > Change contributed by Thomas Adam. > Reference: > https://github.com/tecbot/hive/commit/1e29d88837e4101a76e870a716aadb729437355b#commitcomment-2588350 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3179) HBase Handler doesn't handle NULLs properly
[
https://issues.apache.org/jira/browse/HIVE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574553#comment-13574553
]
Mark Grover commented on HIVE-3179:
---
Running TestHBaseCliDriver tests...
> HBase Handler doesn't handle NULLs properly
> ---
>
> Key: HIVE-3179
> URL: https://issues.apache.org/jira/browse/HIVE-3179
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 0.9.0, 0.10.0
>Reporter: Lars Francke
>Priority: Critical
> Attachments: HIVE-3179.1.patch
>
>
> We found a quite severe issue in the HBase Handler which actually means that
> Hive potentially returns incorrect data if a column has NULL values in HBase
> (which means the cell doesn't even exist)
> In HBase Shell:
> {noformat}
> create 'hive_hbase_test', 'test'
> put 'hive_hbase_test', '1', 'test:c1', 'c1-1'
> put 'hive_hbase_test', '1', 'test:c2', 'c2-1'
> put 'hive_hbase_test', '1', 'test:c3', 'c3-1'
> put 'hive_hbase_test', '2', 'test:c1', 'c1-2'
> {noformat}
> In Hive:
> {noformat}
> DROP TABLE IF EXISTS hive_hbase_test;
> CREATE EXTERNAL TABLE hive_hbase_test (
> id int,
> c1 string,
> c2 string,
> c3 string
> )
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key#s,test:c1#s,test:c2#s,test:c3#s")
> TBLPROPERTIES("hbase.table.name" = "hive_hbase_test");
> hive> select * from hive_hbase_test;
> OK
> 1 c1-1c2-1c3-1
> 2 c1-2NULLNULL
> hive> select c1 from hive_hbase_test;
> c1-1
> c1-2
> hive> select c1, c2 from hive_hbase_test;
> c1-1 c2-1
> c1-2 NULL
> {noformat}
> So far everything is correct but now:
> {noformat}
> hive> select c1, c2, c2 from hive_hbase_test;
> c1-1 c2-1c2-1
> c1-2 NULLc2-1
> {noformat}
> Selecting c2 twice works the first time but the second time we
> actually get the value from the previous row.
> {noformat}
> hive> select c1, c3, c2, c2, c3, c3, c1 from hive_hbase_test;
> c1-1 c3-1c2-1c2-1c3-1c3-1c1-1
> c1-2 NULLNULLc2-1c3-1c3-1c1-2
> {noformat}
> We've narrowed this down to an early initialization of
> {{fieldsInited\[fieldID] = true}} in {{LazyHBaseRow#uncheckedGetField}} and
> we'll try to provide a patch which surely needs review.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3998) Oracle metastore update script will fail when upgrading from 0.9.0 to 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574504#comment-13574504
]
Mark Grover commented on HIVE-3998:
---
Thanks Namit! Could you look at HIVE-3995 as well when you get a chance? It
does the same change to PostgreSQL scripts.
> Oracle metastore update script will fail when upgrading from 0.9.0 to 0.10.0
>
>
> Key: HIVE-3998
> URL: https://issues.apache.org/jira/browse/HIVE-3998
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jarek Jarcec Cecho
> Fix For: 0.11.0
>
> Attachments: oracle_update_issue.patch
>
>
> The problem is in following file {{011-HIVE-3649.oracle.sql}} that contains
> following line:
> {code}
> alter table SDS add IS_STOREDASSUBDIRECTORIES NUMBER(1) NOT NULL;
> {code}
> This query will however fail if the table SDS have at least one row.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3999) Mysql metastore upgrade script will end up with different schema than the full schema load
[
https://issues.apache.org/jira/browse/HIVE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574505#comment-13574505
]
Mark Grover commented on HIVE-3999:
---
Thanks Namit!
> Mysql metastore upgrade script will end up with different schema than the
> full schema load
> --
>
> Key: HIVE-3999
> URL: https://issues.apache.org/jira/browse/HIVE-3999
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jarek Jarcec Cecho
> Fix For: 0.11.0
>
> Attachments: mysql_upgrade_issue.patch
>
>
> I've noticed that the file {{hive-schema-0.10.0.mysql.sql}} is creating table
> SDS with following column:
> {code}
> `IS_STOREDASSUBDIRECTORIES` bit(1) NOT NULL,
> {code}
> However the upgrade script {{011-HIVE-3649.mysql.sql}} will create the column
> differently:
> {code}
> ALTER TABLE `SDS` ADD `IS_STOREDASSUBDIRECTORIES` bit(1) ;
> {code}
> Thus user will get slightly different schema each time - once with "NOT NULL"
> and secondly with "NULL" definition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3999) Mysql metastore upgrade script will end up with different schema than the full schema load
[
https://issues.apache.org/jira/browse/HIVE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574253#comment-13574253
]
Mark Grover commented on HIVE-3999:
---
+1 (non-committer)
> Mysql metastore upgrade script will end up with different schema than the
> full schema load
> --
>
> Key: HIVE-3999
> URL: https://issues.apache.org/jira/browse/HIVE-3999
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jarek Jarcec Cecho
> Attachments: mysql_upgrade_issue.patch
>
>
> I've noticed that the file {{hive-schema-0.10.0.mysql.sql}} is creating table
> SDS with following column:
> {code}
> `IS_STOREDASSUBDIRECTORIES` bit(1) NOT NULL,
> {code}
> However the upgrade script {{011-HIVE-3649.mysql.sql}} will create the column
> differently:
> {code}
> ALTER TABLE `SDS` ADD `IS_STOREDASSUBDIRECTORIES` bit(1) ;
> {code}
> Thus user will get slightly different schema each time - once with "NOT NULL"
> and secondly with "NULL" definition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3998) Oracle metastore update script will fail when upgrading from 0.9.0 to 0.10.0
[
https://issues.apache.org/jira/browse/HIVE-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574251#comment-13574251
]
Mark Grover commented on HIVE-3998:
---
+1 (non-committer)
> Oracle metastore update script will fail when upgrading from 0.9.0 to 0.10.0
>
>
> Key: HIVE-3998
> URL: https://issues.apache.org/jira/browse/HIVE-3998
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Jarek Jarcec Cecho
> Attachments: oracle_update_issue.patch
>
>
> The problem is in following file {{011-HIVE-3649.oracle.sql}} that contains
> following line:
> {code}
> alter table SDS add IS_STOREDASSUBDIRECTORIES NUMBER(1) NOT NULL;
> {code}
> This query will however fail if the table SDS have at least one row.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3995) PostgreSQL upgrade scripts are not valid
[ https://issues.apache.org/jira/browse/HIVE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574249#comment-13574249 ] Mark Grover commented on HIVE-3995: --- +1 (non-committer) Thanks Jarcec! I left a comment (no action required from your side) on Reviewboard that "false" seems to be the right value to put for existing rows. Perhaps, [~gangtimliu] or [~namitjain], who were involved with HIVE-3649 can confirm. I like the new approach of adding the new column as Nullable, updating the existing null values and then setting the column to not-null. The only other approach I can think of is to use a default value (of false) but that seems more risky in the long run. > PostgreSQL upgrade scripts are not valid > > > Key: HIVE-3995 > URL: https://issues.apache.org/jira/browse/HIVE-3995 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Jarek Jarcec Cecho > Attachments: postgre_update_issue.patch, postgre_update_issue.patch > > > I've noticed that scripts for upgrading metastore backed up on PostgreSQL are > not valid. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3995) PostgreSQL upgrade scripts are not valid
[ https://issues.apache.org/jira/browse/HIVE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573110#comment-13573110 ] Mark Grover commented on HIVE-3995: --- Jarcec, thanks for this! The PostgreSQL scripts were not supported until recently, so it's possible the syntax in existing scripts is incorrect. I left some comments on the review. > PostgreSQL upgrade scripts are not valid > > > Key: HIVE-3995 > URL: https://issues.apache.org/jira/browse/HIVE-3995 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Jarek Jarcec Cecho > Attachments: postgre_update_issue.patch > > > I've noticed that scripts for upgrading metastore backed up on PostgreSQL are > not valid. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13570925#comment-13570925 ] Mark Grover commented on HIVE-3725: --- Comments on reviewboard > Add support for pulling HBase columns with prefixes > --- > > Key: HIVE-3725 > URL: https://issues.apache.org/jira/browse/HIVE-3725 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.9.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-3725.1.patch.txt > > > Current HBase Hive integration supports reading many values from the same row > by specifying a column family. And specifying just the column family can pull > in all qualifiers within the family. > We should add in support to be able to specify a prefix for the qualifier and > all columns that start with the prefix would automatically get pulled in. A > wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3810) HiveHistory.log need to replace '\r' with space before writing Entry.value to historyfile
[
https://issues.apache.org/jira/browse/HIVE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-3810:
--
Attachment: HIVE-3810.4.patch
My previous patch (#3) was accidently based off of patch 2 instead of being
based off of trunk. Corrected that in patch 4.
> HiveHistory.log need to replace '\r' with space before writing Entry.value to
> historyfile
> -
>
> Key: HIVE-3810
> URL: https://issues.apache.org/jira/browse/HIVE-3810
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: qiangwang
>Assignee: Mark Grover
>Priority: Minor
> Attachments: HIVE-3810.1.patch, HIVE-3810.2.patch, HIVE-3810.3.patch,
> HIVE-3810.4.patch
>
>
> HiveHistory.log will replace '\n' with space before writing Entry.value to
> history file:
> val = val.replace('\n', ' ');
> but HiveHistory.parseHiveHistory use BufferedReader.readLine which takes
> '\n', '\r', '\r\n' as line delimiter to parse history file
> if val contains '\r', there is a high possibility that HiveHistory.parseLine
> will fail, in which case usually RecordTypes.valueOf(recType) will throw
> exception 'java.lang.IllegalArgumentException'
> HiveHistory.log need to replace '\r' with space as well:
> val = val.replace('\n', ' ');
> changed to
> val = val.replaceAll("\r|\n", " ");
> or
> val = val.replace('\r', ' ').replace('\n', ' ');
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3810) HiveHistory.log need to replace '\r' with space before writing Entry.value to historyfile
[
https://issues.apache.org/jira/browse/HIVE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-3810:
--
Attachment: HIVE-3810.3.patch
> HiveHistory.log need to replace '\r' with space before writing Entry.value to
> historyfile
> -
>
> Key: HIVE-3810
> URL: https://issues.apache.org/jira/browse/HIVE-3810
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: qiangwang
>Assignee: Mark Grover
>Priority: Minor
> Attachments: HIVE-3810.1.patch, HIVE-3810.2.patch, HIVE-3810.3.patch
>
>
> HiveHistory.log will replace '\n' with space before writing Entry.value to
> history file:
> val = val.replace('\n', ' ');
> but HiveHistory.parseHiveHistory use BufferedReader.readLine which takes
> '\n', '\r', '\r\n' as line delimiter to parse history file
> if val contains '\r', there is a high possibility that HiveHistory.parseLine
> will fail, in which case usually RecordTypes.valueOf(recType) will throw
> exception 'java.lang.IllegalArgumentException'
> HiveHistory.log need to replace '\r' with space as well:
> val = val.replace('\n', ' ');
> changed to
> val = val.replaceAll("\r|\n", " ");
> or
> val = val.replace('\r', ' ').replace('\n', ' ');
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3810) HiveHistory.log need to replace '\r' with space before writing Entry.value to historyfile
[
https://issues.apache.org/jira/browse/HIVE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-3810:
--
Status: Patch Available (was: Reopened)
> HiveHistory.log need to replace '\r' with space before writing Entry.value to
> historyfile
> -
>
> Key: HIVE-3810
> URL: https://issues.apache.org/jira/browse/HIVE-3810
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: qiangwang
>Assignee: Mark Grover
>Priority: Minor
> Attachments: HIVE-3810.1.patch, HIVE-3810.2.patch, HIVE-3810.3.patch
>
>
> HiveHistory.log will replace '\n' with space before writing Entry.value to
> history file:
> val = val.replace('\n', ' ');
> but HiveHistory.parseHiveHistory use BufferedReader.readLine which takes
> '\n', '\r', '\r\n' as line delimiter to parse history file
> if val contains '\r', there is a high possibility that HiveHistory.parseLine
> will fail, in which case usually RecordTypes.valueOf(recType) will throw
> exception 'java.lang.IllegalArgumentException'
> HiveHistory.log need to replace '\r' with space as well:
> val = val.replace('\n', ' ');
> changed to
> val = val.replaceAll("\r|\n", " ");
> or
> val = val.replace('\r', ' ').replace('\n', ' ');
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3810) HiveHistory.log need to replace '\r' with space before writing Entry.value to historyfile
[
https://issues.apache.org/jira/browse/HIVE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569937#comment-13569937
]
Mark Grover commented on HIVE-3810:
---
Thanks, Edward for the review!
Your comment made me think about using {{System.getProperty("line.separator")}}
instead.
Let me post a new patch with that. I am not entirely sure how to test it (apart
from running the unit tests). If you have any ideas, please let me know.
> HiveHistory.log need to replace '\r' with space before writing Entry.value to
> historyfile
> -
>
> Key: HIVE-3810
> URL: https://issues.apache.org/jira/browse/HIVE-3810
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: qiangwang
>Assignee: Mark Grover
>Priority: Minor
> Attachments: HIVE-3810.1.patch, HIVE-3810.2.patch, HIVE-3810.3.patch
>
>
> HiveHistory.log will replace '\n' with space before writing Entry.value to
> history file:
> val = val.replace('\n', ' ');
> but HiveHistory.parseHiveHistory use BufferedReader.readLine which takes
> '\n', '\r', '\r\n' as line delimiter to parse history file
> if val contains '\r', there is a high possibility that HiveHistory.parseLine
> will fail, in which case usually RecordTypes.valueOf(recType) will throw
> exception 'java.lang.IllegalArgumentException'
> HiveHistory.log need to replace '\r' with space as well:
> val = val.replace('\n', ' ');
> changed to
> val = val.replaceAll("\r|\n", " ");
> or
> val = val.replace('\r', ' ').replace('\n', ' ');
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3932) Hive release tarballs don't contain PostgreSQL metastore scripts
[ https://issues.apache.org/jira/browse/HIVE-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569934#comment-13569934 ] Mark Grover commented on HIVE-3932: --- Thanks, Ashutosh, for committing this! > Hive release tarballs don't contain PostgreSQL metastore scripts > > > Key: HIVE-3932 > URL: https://issues.apache.org/jira/browse/HIVE-3932 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3932.1.patch > > > By means of HIVE-2552, we decided to not include Hive metastore upgrade > scripts for PostgreSQL in Hive release tarballs. This was done primarily > because the scripts were incomplete at that time. However, since then, > starting at least 0.10.0 (maybe 0.9?) PostgreSQL scripts are now being > maintained and any changes to the metastore are reflected to them > (HIVE-2529). Consequently, there is no reason to not release PostgreSQL > scripts. This JIRA plans to disable/remove/ ignore the "include.postgres" > property in build.xml that enables the exclusion of PostgreSQL scripts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3918) Normalize more CRLF line endings
[ https://issues.apache.org/jira/browse/HIVE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569933#comment-13569933 ] Mark Grover commented on HIVE-3918: --- Thanks, Ashutosh, for committing this! > Normalize more CRLF line endings > > > Key: HIVE-3918 > URL: https://issues.apache.org/jira/browse/HIVE-3918 > Project: Hive > Issue Type: Bug >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3918.1.patch > > > While I tried to fix files that had incompatible line endings in HIVE-3858, I > missed some more files. It was most likely because HIVE-3858 was tested by me > on Mac OS X but when I started using the github repository on a ubuntu box, I > saw there are some more files in the Hive repo that have incorrect line > endings. These files get added to the index automatically when a change is > made to the local git repo. Once these cosmetic changes get committed, we > would have gotten rid of all windows/old Mac OS style line endings in our > Hive repository. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2693) Add DECIMAL data type
[ https://issues.apache.org/jira/browse/HIVE-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569234#comment-13569234 ] Mark Grover commented on HIVE-2693: --- Thanks for committing this, Carl! FYI, I have updated the wiki pages with info about Decimal type. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL > Add DECIMAL data type > - > > Key: HIVE-2693 > URL: https://issues.apache.org/jira/browse/HIVE-2693 > Project: Hive > Issue Type: New Feature > Components: Query Processor, Types >Affects Versions: 0.10.0 >Reporter: Carl Steinbach >Assignee: Prasad Mujumdar > Fix For: 0.11.0 > > Attachments: 2693_7.patch, 2693_8.patch, 2693_fix_all_tests1.patch, > HIVE-2693-10.patch, HIVE-2693-11.patch, HIVE-2693-12-SortableSerDe.patch, > HIVE-2693-13.patch, HIVE-2693-14.patch, HIVE-2693-15.patch, > HIVE-2693-16.patch, HIVE-2693-17.patch, HIVE-2693-18.patch, > HIVE-2693-19.patch, HIVE-2693-1.patch.txt, HIVE-2693-20.patch, > HIVE-2693-21.patch, HIVE-2693-22.patch, HIVE-2693-23.patch, > HIVE-2693-all.patch, HIVE-2693.D7683.1.patch, HIVE-2693-fix.patch, > HIVE-2693.patch, HIVE-2693-take3.patch, HIVE-2693-take4.patch > > > Add support for the DECIMAL data type. HIVE-2272 (TIMESTAMP) provides a nice > template for how to do this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3976) Support specifying scale and precision with Hive decimal type
Mark Grover created HIVE-3976:
-
Summary: Support specifying scale and precision with Hive decimal
type
Key: HIVE-3976
URL: https://issues.apache.org/jira/browse/HIVE-3976
Project: Hive
Issue Type: Improvement
Components: Query Processor, Types
Reporter: Mark Grover
HIVE-2693 introduced support for Decimal datatype in Hive. However, the current
implementation has unlimited precision and provides no way to specify precision
and scale when creating the table.
For example, MySQL allows users to specify scale and precision of the decimal
datatype when creating the table:
{code}
CREATE TABLE numbers (a DECIMAL(20,2));
{code}
Hive should support something similar too.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3975) Hive JIRA still shows 0.10 as unreleased in "Affects Version/s" dropdown
Mark Grover created HIVE-3975: - Summary: Hive JIRA still shows 0.10 as unreleased in "Affects Version/s" dropdown Key: HIVE-3975 URL: https://issues.apache.org/jira/browse/HIVE-3975 Project: Hive Issue Type: Bug Components: Website Affects Versions: 0.10.0 Reporter: Mark Grover Fix For: 0.10.0 The JIRA still shows 0.10.0 as unreleased when selecting "Affects Version/s" and "Fix Version/s". Now that Hive 0.10.0 has been released, can we please change that? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3850) hour() function returns 12 hour clock value when using timestamp datatype
[
https://issues.apache.org/jira/browse/HIVE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13567763#comment-13567763
]
Mark Grover commented on HIVE-3850:
---
It's all good, we were all new at some point:-)
Here is what I would do for the tests:
Developer Guide is a really good link to read on how unit tests work in Hive:
https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-Unittestsanddebugging
Practically, unit tests take a long time to run on especially if you run it on
1 node (like me), so for a minor change like this, I run only the tests that
are impacted.
To figure that out what's impacted, I would do something like:
{code}
cd ql/src/test/queries
grep -ri "hour" *
{code}
For results under clientpositive, I would run something like:
{code}
ant test -Dtestcase=TestCliDriver -Dqfile=
{code}
For results under clientnegative, I would run something like:
{code}
ant test TestNegativeCliDriver -Dqfile=
{code}
If any of the above tests fail, it may legitimately be because of the bug fix,
in which case, you would want to update the test output. You can do that by
using {{-Doverwrite=true}} at the end of your commands. If the outputs, do need
to get updated, please include those diffs in the patch as well.
The committers would usually run the whole suite of tests (some times on faster
clusters), so don't be paranoid about testing all tests but of course, do you
due-diligence about finding all tests that may be impacted and making sure they
run fine.
Hope this makes sense. If you have any further questions, please don't hesitate
to ask. And, thanks for working on this patch!
> hour() function returns 12 hour clock value when using timestamp datatype
> -
>
> Key: HIVE-3850
> URL: https://issues.apache.org/jira/browse/HIVE-3850
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 0.9.0, 0.10.0
>Reporter: Pieterjan Vriends
> Fix For: 0.11.0
>
> Attachments: HIVE-3850.patch.txt
>
>
> Apparently UDFHour.java does have two evaluate() functions. One that does
> accept a Text object as parameter and one that does use a TimeStampWritable
> object as parameter. The first function does return the value of
> Calendar.HOUR_OF_DAY and the second one of Calendar.HOUR. In the
> documentation I couldn't find any information on the overload of the
> evaluation function. I did spent quite some time finding out why my statement
> didn't return a 24 hour clock value.
> Shouldn't both functions return the same?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13567446#comment-13567446 ] Mark Grover commented on HIVE-3951: --- Code review at https://reviews.apache.org/r/9173/ > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3951: -- Attachment: HIVE-3951.1.patch Patch attached, adds support for decimal columns regex serde. Added test. > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3951) Allow Decimal type columns in Regex Serde
[ https://issues.apache.org/jira/browse/HIVE-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3951: -- Status: Patch Available (was: Open) > Allow Decimal type columns in Regex Serde > - > > Key: HIVE-3951 > URL: https://issues.apache.org/jira/browse/HIVE-3951 > Project: Hive > Issue Type: New Feature > Components: Serializers/Deserializers >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover > Fix For: 0.11.0 > > Attachments: HIVE-3951.1.patch > > > Decimal type in Hive was recently added by HIVE-2693. We should allow users > to create tables with decimal type columns when using Regex Serde. > HIVE-3004 did something similar for other primitive types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3850) hour() function returns 12 hour clock value when using timestamp datatype
[ https://issues.apache.org/jira/browse/HIVE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3850: -- Affects Version/s: 0.10.0 Fix Version/s: 0.11.0 > hour() function returns 12 hour clock value when using timestamp datatype > - > > Key: HIVE-3850 > URL: https://issues.apache.org/jira/browse/HIVE-3850 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.9.0, 0.10.0 >Reporter: Pieterjan Vriends > Fix For: 0.11.0 > > Attachments: HIVE-3850.patch.txt > > > Apparently UDFHour.java does have two evaluate() functions. One that does > accept a Text object as parameter and one that does use a TimeStampWritable > object as parameter. The first function does return the value of > Calendar.HOUR_OF_DAY and the second one of Calendar.HOUR. In the > documentation I couldn't find any information on the overload of the > evaluation function. I did spent quite some time finding out why my statement > didn't return a 24 hour clock value. > Shouldn't both functions return the same? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-3850) hour() function returns 12 hour clock value when using timestamp datatype
[ https://issues.apache.org/jira/browse/HIVE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover reopened HIVE-3850: --- The change wasn't committed, re-opening the JIRA. > hour() function returns 12 hour clock value when using timestamp datatype > - > > Key: HIVE-3850 > URL: https://issues.apache.org/jira/browse/HIVE-3850 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.9.0, 0.10.0 >Reporter: Pieterjan Vriends > Fix For: 0.11.0 > > Attachments: HIVE-3850.patch.txt > > > Apparently UDFHour.java does have two evaluate() functions. One that does > accept a Text object as parameter and one that does use a TimeStampWritable > object as parameter. The first function does return the value of > Calendar.HOUR_OF_DAY and the second one of Calendar.HOUR. In the > documentation I couldn't find any information on the overload of the > evaluation function. I did spent quite some time finding out why my statement > didn't return a 24 hour clock value. > Shouldn't both functions return the same? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3850) hour() function returns 12 hour clock value when using timestamp datatype
[ https://issues.apache.org/jira/browse/HIVE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13567356#comment-13567356 ] Mark Grover commented on HIVE-3850: --- Patch looks good to me. Usually, I would ask for unit tests to be added with any change but given that it's a trivial change, I would be ok without new tests. We should, however, make sure we update the existing unit tests if needed. Did you get a chance to run the unit tests (atleast the ones that use hour UDF) and make sure no changes are required in their output? > hour() function returns 12 hour clock value when using timestamp datatype > - > > Key: HIVE-3850 > URL: https://issues.apache.org/jira/browse/HIVE-3850 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.9.0 >Reporter: Pieterjan Vriends > Attachments: HIVE-3850.patch.txt > > > Apparently UDFHour.java does have two evaluate() functions. One that does > accept a Text object as parameter and one that does use a TimeStampWritable > object as parameter. The first function does return the value of > Calendar.HOUR_OF_DAY and the second one of Calendar.HOUR. In the > documentation I couldn't find any information on the overload of the > evaluation function. I did spent quite some time finding out why my statement > didn't return a 24 hour clock value. > Shouldn't both functions return the same? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3850) hour() function returns 12 hour clock value when using timestamp datatype
[ https://issues.apache.org/jira/browse/HIVE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13567355#comment-13567355 ] Mark Grover commented on HIVE-3850: --- For completeness, review is at: https://reviews.apache.org/r/9171/ > hour() function returns 12 hour clock value when using timestamp datatype > - > > Key: HIVE-3850 > URL: https://issues.apache.org/jira/browse/HIVE-3850 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.9.0 >Reporter: Pieterjan Vriends > Attachments: HIVE-3850.patch.txt > > > Apparently UDFHour.java does have two evaluate() functions. One that does > accept a Text object as parameter and one that does use a TimeStampWritable > object as parameter. The first function does return the value of > Calendar.HOUR_OF_DAY and the second one of Calendar.HOUR. In the > documentation I couldn't find any information on the overload of the > evaluation function. I did spent quite some time finding out why my statement > didn't return a 24 hour clock value. > Shouldn't both functions return the same? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3957) Add pseudo-BNF grammar for RCFile to Javadoc
[ https://issues.apache.org/jira/browse/HIVE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3957: -- Attachment: HIVE-3957.2.patch Rendered javadoc looks much better now. > Add pseudo-BNF grammar for RCFile to Javadoc > > > Key: HIVE-3957 > URL: https://issues.apache.org/jira/browse/HIVE-3957 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover >Priority: Minor > Fix For: 0.11.0 > > Attachments: HIVE-3957.1.patch, HIVE-3957.2.patch > > > Carl Steinbach did wrote some nice pseudo-BNF grammar for RCFile. I think it > would rad to put it in the Javadoc to benefit other developers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3957) Add pseudo-BNF grammar for RCFile to Javadoc
[ https://issues.apache.org/jira/browse/HIVE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565156#comment-13565156 ] Mark Grover commented on HIVE-3957: --- Some of the comments are being garbled in Javadoc. New patch coming soon. > Add pseudo-BNF grammar for RCFile to Javadoc > > > Key: HIVE-3957 > URL: https://issues.apache.org/jira/browse/HIVE-3957 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover >Priority: Minor > Fix For: 0.11.0 > > Attachments: HIVE-3957.1.patch > > > Carl Steinbach did wrote some nice pseudo-BNF grammar for RCFile. I think it > would rad to put it in the Javadoc to benefit other developers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3957) Add pseudo-BNF grammar for RCFile to Javadoc
[ https://issues.apache.org/jira/browse/HIVE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3957: -- Status: Patch Available (was: Open) > Add pseudo-BNF grammar for RCFile to Javadoc > > > Key: HIVE-3957 > URL: https://issues.apache.org/jira/browse/HIVE-3957 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover >Priority: Minor > Fix For: 0.11.0 > > Attachments: HIVE-3957.1.patch > > > Carl Steinbach did wrote some nice pseudo-BNF grammar for RCFile. I think it > would rad to put it in the Javadoc to benefit other developers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3957) Add pseudo-BNF grammar for RCFile to Javadoc
[ https://issues.apache.org/jira/browse/HIVE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Grover updated HIVE-3957: -- Attachment: HIVE-3957.1.patch > Add pseudo-BNF grammar for RCFile to Javadoc > > > Key: HIVE-3957 > URL: https://issues.apache.org/jira/browse/HIVE-3957 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.10.0 >Reporter: Mark Grover >Assignee: Mark Grover >Priority: Minor > Fix For: 0.11.0 > > Attachments: HIVE-3957.1.patch > > > Carl Steinbach did wrote some nice pseudo-BNF grammar for RCFile. I think it > would rad to put it in the Javadoc to benefit other developers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-3957) Add pseudo-BNF grammar for RCFile to Javadoc
Mark Grover created HIVE-3957: - Summary: Add pseudo-BNF grammar for RCFile to Javadoc Key: HIVE-3957 URL: https://issues.apache.org/jira/browse/HIVE-3957 Project: Hive Issue Type: Bug Components: Documentation Affects Versions: 0.10.0 Reporter: Mark Grover Assignee: Mark Grover Priority: Minor Fix For: 0.11.0 Carl Steinbach did wrote some nice pseudo-BNF grammar for RCFile. I think it would rad to put it in the Javadoc to benefit other developers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3810) HiveHistory.log need to replace '\r' with space before writing Entry.value to historyfile
[
https://issues.apache.org/jira/browse/HIVE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13564975#comment-13564975
]
Mark Grover commented on HIVE-3810:
---
Review at https://reviews.apache.org/r/9129/
> HiveHistory.log need to replace '\r' with space before writing Entry.value to
> historyfile
> -
>
> Key: HIVE-3810
> URL: https://issues.apache.org/jira/browse/HIVE-3810
> Project: Hive
> Issue Type: Bug
> Components: Logging
>Reporter: qiangwang
>Assignee: Mark Grover
>Priority: Minor
> Attachments: HIVE-3810.1.patch, HIVE-3810.2.patch
>
>
> HiveHistory.log will replace '\n' with space before writing Entry.value to
> history file:
> val = val.replace('\n', ' ');
> but HiveHistory.parseHiveHistory use BufferedReader.readLine which takes
> '\n', '\r', '\r\n' as line delimiter to parse history file
> if val contains '\r', there is a high possibility that HiveHistory.parseLine
> will fail, in which case usually RecordTypes.valueOf(recType) will throw
> exception 'java.lang.IllegalArgumentException'
> HiveHistory.log need to replace '\r' with space as well:
> val = val.replace('\n', ' ');
> changed to
> val = val.replaceAll("\r|\n", " ");
> or
> val = val.replace('\r', ' ').replace('\n', ' ');
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
