Re: Broken Hive docs -- links not working

[Carl Steinbach]

The best way to get this resolved is to file an ASFINFRA JIRA ticket. It
doesn't look like any of us have the privileges necessary to stop the
autoexport job, so the ball is clearly in INFRA's court.

Thanks.

Carl


On Mon, Sep 2, 2013 at 3:37 PM, Lefty Leverenz wrote:

> Edward Capriolo talked about broken links in the Hive wiki in another
> thread on the users list ("What is this windowing query supposed to do?")
> so I'm moving the discussion here.
>
> Recap:
>
> Edward:
>
> > Most of the links here:
> > https://cwiki.apache.org/Hive/languagemanual.html
> >
> > are broken...
> > What gives?
> > The language manual used to be fairly useful, now it is in major
> dis-array.
> >
>
> Lefty:
>
> > Once again the /confluence/display version of the wiki is fine while the
> > non-display version is glitchy.  First the {code} tags gave invisible
> > sample code, and now some of the doc links don't work -- but they're fine
> > here:  https://cwiki.apache.org/confluence/display/Hive/LanguageManual.
> >
> > Can anyone explain this, or better yet solve it?  Until it's solved,
> > should we include a link to the display version at the top of each
> wikidoc?
> >
>
> Edward:
>
> > The problem is if you follow our site from the main page you find the
> > broken docs, not the confluence ones.
> >
>
> New reply from Lefty:  Do you mean the Wiki tab?  When I follow it, I go to
> the "display" version at
> https://cwiki.apache.org/confluence/display/Hive/Home, not the broken
> version at https://cwiki.apache.org/Hive/home.html.
>
> Another problem is that there's no link to the wiki in the menu under
> Documentation.  That can be fixed fairly easily, so I'll open a JIRA.
>
> Spot check for broken links in the non-display
> version
> :
>
>- Getting Started:  404 Not Found
>(https://cwiki.apache.org/Hive/gettingstarted.html)
>- Tutorial:  404 Not Found
>(https://cwiki.apache.org/Hive/tutorial.html)
>- Language Manual:  okay
>(https://cwiki.apache.org/Hive/languagemanual.html)
>- Operators and Functions: 404 Not Found
>(https://cwiki.apache.org/Hive/operatorsandfunctions.html)
>- Web Interface:  404 Not Found
>(https://cwiki.apache.org/Hive/hivewebinterface.html)
>- Hive Client: okay
>(https://cwiki.apache.org/Hive/hiveclient.html)
>- HiveServer2 Clients:  okay
>(https://cwiki.apache.org/Hive/hiveserver2-clients.html)
>- Change Log:  404 Not Found
>(https://cwiki.apache.org/Hive/hivechangelog.html)
>- Avro SerDe:  okay
>(https://cwiki.apache.org/Hive/avroserde.html)
>
> So much for my hope of finding a pattern.  Now to check how the links are
> marked up:
>
> h1. General Information about Hive
>
> * [Getting Started|GettingStarted]
> [...]
>
> h1. User Documentation
>
> * [Hive Tutorial|Tutorial]
> * [HiveQL Language Manual (Queries, DML, DDL, and CLI)|LanguageManual]
> * [Hive Operators and Functions|OperatorsAndFunctions]
> * [Hive Web Interface|HiveWebInterface]
> * [Hive Client (JDBC, ODBC, Thrift, etc)|HiveClient]
> * [HiveServer2 Client|
> https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients]
> * [Hive Change Log|HiveChangeLog]
> * [Avro SerDe|AvroSerDe]
>
>
> Again, nothing obvious.  I think the use of a URL for HS2 Clients was just
> to get around the space in the title, but there's probably another way to
> do it -- I'll check my notes and fix it if I can.
>
> I'll check a few more later, but this doesn't seem to be fruitful
> debugging.  It isn't a matter of new vs. old docs (both Hive Client and HS2
> Clients are okay).  All of these links work fine in the
> "display"
> version of the wiki.  Most of them are top-level files in the
> hierarchy<
> https://cwiki.apache.org/confluence/pages/listpages-dirview.action?key=Hive&openId=27362069#selectedPageInHierarchy
> >,
> except for Operators and Functions (a broken link), HiveServer2 Clients (a
> link that works), and Hive Change Log (broken) and those are all children
> of the Home page.
>
> It's a mystery.
>
> -- Lefty
>

Broken Hive docs -- links not working

[Lefty Leverenz]

Edward Capriolo talked about broken links in the Hive wiki in another
thread on the users list ("What is this windowing query supposed to do?")
so I'm moving the discussion here.

Recap:

Edward:

> Most of the links here:
> https://cwiki.apache.org/Hive/languagemanual.html
>
> are broken...
> What gives?
> The language manual used to be fairly useful, now it is in major dis-array.
>

Lefty:

> Once again the /confluence/display version of the wiki is fine while the
> non-display version is glitchy.  First the {code} tags gave invisible
> sample code, and now some of the doc links don't work -- but they're fine
> here:  https://cwiki.apache.org/confluence/display/Hive/LanguageManual.
>
> Can anyone explain this, or better yet solve it?  Until it's solved,
> should we include a link to the display version at the top of each wikidoc?
>

Edward:

> The problem is if you follow our site from the main page you find the
> broken docs, not the confluence ones.
>

New reply from Lefty:  Do you mean the Wiki tab?  When I follow it, I go to
the "display" version at
https://cwiki.apache.org/confluence/display/Hive/Home, not the broken
version at https://cwiki.apache.org/Hive/home.html.

Another problem is that there's no link to the wiki in the menu under
Documentation.  That can be fixed fairly easily, so I'll open a JIRA.

Spot check for broken links in the non-display
version
:

   - Getting Started:  404 Not Found
   (https://cwiki.apache.org/Hive/gettingstarted.html)
   - Tutorial:  404 Not Found
   (https://cwiki.apache.org/Hive/tutorial.html)
   - Language Manual:  okay
   (https://cwiki.apache.org/Hive/languagemanual.html)
   - Operators and Functions: 404 Not Found
   (https://cwiki.apache.org/Hive/operatorsandfunctions.html)
   - Web Interface:  404 Not Found
   (https://cwiki.apache.org/Hive/hivewebinterface.html)
   - Hive Client: okay
   (https://cwiki.apache.org/Hive/hiveclient.html)
   - HiveServer2 Clients:  okay
   (https://cwiki.apache.org/Hive/hiveserver2-clients.html)
   - Change Log:  404 Not Found
   (https://cwiki.apache.org/Hive/hivechangelog.html)
   - Avro SerDe:  okay
   (https://cwiki.apache.org/Hive/avroserde.html)

So much for my hope of finding a pattern.  Now to check how the links are
marked up:

h1. General Information about Hive

* [Getting Started|GettingStarted]
[...]

h1. User Documentation

* [Hive Tutorial|Tutorial]
* [HiveQL Language Manual (Queries, DML, DDL, and CLI)|LanguageManual]
* [Hive Operators and Functions|OperatorsAndFunctions]
* [Hive Web Interface|HiveWebInterface]
* [Hive Client (JDBC, ODBC, Thrift, etc)|HiveClient]
* [HiveServer2 Client|
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients]
* [Hive Change Log|HiveChangeLog]
* [Avro SerDe|AvroSerDe]


Again, nothing obvious.  I think the use of a URL for HS2 Clients was just
to get around the space in the title, but there's probably another way to
do it -- I'll check my notes and fix it if I can.

I'll check a few more later, but this doesn't seem to be fruitful
debugging.  It isn't a matter of new vs. old docs (both Hive Client and HS2
Clients are okay).  All of these links work fine in the
"display"
version of the wiki.  Most of them are top-level files in the
hierarchy,
except for Operators and Functions (a broken link), HiveServer2 Clients (a
link that works), and Hive Change Log (broken) and those are all children
of the Home page.

It's a mystery.

-- Lefty

Re: DISCUSS: Hive language manual to be source control managed

[Edward Capriolo]

I think that pages with only syntax that do not reference a source table or
show results are not very useful to me.

This page is nice:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView

This page is not nearly as useful:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VirtualColumns

I just do not think we can get a consistent feel and style.

Also for reference, I think the lateral view syntax is out of date because
we no longer require the AS.


On Mon, Sep 2, 2013 at 5:05 PM, Lars Francke  wrote:

> > The page that strikes me as alarmingly poor is the:
> > https://cwiki.apache.org/Hive/languagemanual.html
>
> The correct link is:
> 
>
> For some reason there is an "Auto Export" Plugin which exports all
> Confluence pages as static HTML. That process is apparently broken.
> Unfortunately those pages appear as the top results in Google.
>
> > The language manual is in very poor shape at the moment with broken
> links,
> > incorrect content, incomplete content, and poor coverage of the actual
> > languages. IMHO the attempts to crowd-source this documentation has
> failed.
> > Having a good concise language manual is critical to the success and
> > adoption of hive.
>
> I agree. A few people try to keep up the documentation but as long as
> patches are accepted without any documentation this situation won't
> improve no matter the format of the documentation. Often not even the
> JIRA issues even mention the actual changes, you have to dig in the
> source code to find the new features/settings. I assume that a lot of
> communication happens offline.
>
> So one big step forward would be to not commit any feature changes
> without documentation in the Wiki or at least a clear and up to date
> description of the change in the JIRA.
>
> Cheers,
> Lars
>

Re: What is this windowing query supposed to do?

[Lefty Leverenz]

> The problem is if you follow our site from the main page you find the
broken docs, not the confluence ones.

Let's take this to a separate thread, since it isn't about the windowing
query.

-- Lefty


On Mon, Sep 2, 2013 at 11:44 AM, Edward Capriolo wrote:

> The problem is if you follow our site from the main page you find the
> broken docs, not the confluence ones.
>
>
> On Mon, Sep 2, 2013 at 4:21 AM, Lefty Leverenz wrote:
>
>> Once again the /confluence/display version of the wiki is fine while the
>> non-display version is glitchy.  First the {code} tags gave invisible
>> sample code, and now some of the doc links don't work -- but they're fine
>> here:  https://cwiki.apache.org/confluence/display/Hive/LanguageManual.
>>
>> Can anyone explain this, or better yet solve it?  Until it's solved,
>> should we include a link to the display version at the top of each wikidoc?
>>  (And I suppose that's the royal "we" unless someone else volunteers.)
>>
>> For the windowing 
>> doc,
>> I've added a link to Harish's PDF file in the first paragraph:
>>
>>- See "Windowing Specifications in 
>> HQL"
>>(attached to HIVE-4197)
>>for details.
>>
>> Happy Labor Day, everyone.  -- Lefty
>>
>>
>>
>> On Sun, Sep 1, 2013 at 8:36 PM, Harish Butani wrote:
>>
>>> In your 3rd query the OVER clause is being ignored because there is no
>>> associated Aggregation Function.
>>> This has been fixed in HIVE-4214. The query fails with the following
>>> error:
>>> 'cannot recognize input near 'range' 'OVER' '(' in select expression'
>>>
>>> The Windowing behavior in Hive is documented in HIVE-4197
>>>
>>> https://issues.apache.org/jira/secure/attachment/12575830/WindowingSpecification.pdf
>>>
>>> On Sep 1, 2013, at 2:04 PM, Edward Capriolo 
>>> wrote:
>>>
>>> hive> select * from field_goal;
>>> OK
>>> tynes44miss
>>> tynes40miss
>>> tynes33make
>>> pete20make
>>> pete21make
>>> arnold56make
>>> arnold22make
>>> arnold45miss
>>> Time taken: 0.04 seconds, Fetched: 8 row(s)
>>>
>>> This makes sense:
>>> hive> select name,avg(range) OVER (PARTITION by name ORDER by range)
>>> FROM field_goal  ;
>>> arnold22.0
>>> arnold33.5
>>> arnold41.0
>>> pete20.0
>>> pete20.5
>>> tynes33.0
>>> tynes36.5
>>> tynes39.0
>>> Time taken: 4.14 seconds, Fetched: 8 row(s)
>>>
>>> but what is this?
>>> hive> select name,range OVER (PARTITION by name ORDER by range) FROM
>>> field_goal  ;
>>> tynes44
>>> tynes40
>>> tynes33
>>> pete20
>>> pete21
>>> arnold56
>>> arnold22
>>> arnold45
>>> Time taken: 3.709 seconds, Fetched: 8 row(s)
>>>
>>> It would be great if the
>>> https://cwiki.apache.org/Hive/languagemanual-windowingandanalytics.html#LanguageManualWindowingAndAnalytics-PARTITIONBYwithpartitioning%252CORDERBY%252Candwindowspecification
>>>
>>>
>>> Most of the links here:
>>> https://cwiki.apache.org/Hive/languagemanual.html
>>>
>>> are broken...
>>> What gives?
>>> The language manual used to be fairly useful, now it is in major
>>> dis-array.
>>>
>>>
>>>
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>>
>

Re: DISCUSS: Hive language manual to be source control managed

[Lars Francke]

> The page that strikes me as alarmingly poor is the:
> https://cwiki.apache.org/Hive/languagemanual.html

The correct link is:


For some reason there is an "Auto Export" Plugin which exports all
Confluence pages as static HTML. That process is apparently broken.
Unfortunately those pages appear as the top results in Google.

> The language manual is in very poor shape at the moment with broken links,
> incorrect content, incomplete content, and poor coverage of the actual
> languages. IMHO the attempts to crowd-source this documentation has failed.
> Having a good concise language manual is critical to the success and
> adoption of hive.

I agree. A few people try to keep up the documentation but as long as
patches are accepted without any documentation this situation won't
improve no matter the format of the documentation. Often not even the
JIRA issues even mention the actual changes, you have to dig in the
source code to find the new features/settings. I assume that a lot of
communication happens offline.

So one big step forward would be to not commit any feature changes
without documentation in the Wiki or at least a clear and up to date
description of the change in the JIRA.

Cheers,
Lars

Re: What is this windowing query supposed to do?

[Edward Capriolo]

The problem is if you follow our site from the main page you find the
broken docs, not the confluence ones.


On Mon, Sep 2, 2013 at 4:21 AM, Lefty Leverenz wrote:

> Once again the /confluence/display version of the wiki is fine while the
> non-display version is glitchy.  First the {code} tags gave invisible
> sample code, and now some of the doc links don't work -- but they're fine
> here:  https://cwiki.apache.org/confluence/display/Hive/LanguageManual.
>
> Can anyone explain this, or better yet solve it?  Until it's solved,
> should we include a link to the display version at the top of each wikidoc?
>  (And I suppose that's the royal "we" unless someone else volunteers.)
>
> For the windowing 
> doc,
> I've added a link to Harish's PDF file in the first paragraph:
>
>- See "Windowing Specifications in 
> HQL"
>(attached to HIVE-4197)
>for details.
>
> Happy Labor Day, everyone.  -- Lefty
>
>
>
> On Sun, Sep 1, 2013 at 8:36 PM, Harish Butani wrote:
>
>> In your 3rd query the OVER clause is being ignored because there is no
>> associated Aggregation Function.
>> This has been fixed in HIVE-4214. The query fails with the following
>> error:
>> 'cannot recognize input near 'range' 'OVER' '(' in select expression'
>>
>> The Windowing behavior in Hive is documented in HIVE-4197
>>
>> https://issues.apache.org/jira/secure/attachment/12575830/WindowingSpecification.pdf
>>
>> On Sep 1, 2013, at 2:04 PM, Edward Capriolo 
>> wrote:
>>
>> hive> select * from field_goal;
>> OK
>> tynes44miss
>> tynes40miss
>> tynes33make
>> pete20make
>> pete21make
>> arnold56make
>> arnold22make
>> arnold45miss
>> Time taken: 0.04 seconds, Fetched: 8 row(s)
>>
>> This makes sense:
>> hive> select name,avg(range) OVER (PARTITION by name ORDER by range) FROM
>> field_goal  ;
>> arnold22.0
>> arnold33.5
>> arnold41.0
>> pete20.0
>> pete20.5
>> tynes33.0
>> tynes36.5
>> tynes39.0
>> Time taken: 4.14 seconds, Fetched: 8 row(s)
>>
>> but what is this?
>> hive> select name,range OVER (PARTITION by name ORDER by range) FROM
>> field_goal  ;
>> tynes44
>> tynes40
>> tynes33
>> pete20
>> pete21
>> arnold56
>> arnold22
>> arnold45
>> Time taken: 3.709 seconds, Fetched: 8 row(s)
>>
>> It would be great if the
>> https://cwiki.apache.org/Hive/languagemanual-windowingandanalytics.html#LanguageManualWindowingAndAnalytics-PARTITIONBYwithpartitioning%252CORDERBY%252Candwindowspecification
>>
>>
>> Most of the links here:
>> https://cwiki.apache.org/Hive/languagemanual.html
>>
>> are broken...
>> What gives?
>> The language manual used to be fairly useful, now it is in major
>> dis-array.
>>
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>

Re: DISCUSS: Hive language manual to be source control managed

[Edward Capriolo]

I understand your concerns, but
"Last year I converted the entire Language Manual to xdocs, but the project
died from lack of community interest."
I believe I converted a majority of the wiki before you did
https://github.com/mislam77/hive-li/blob/master/docs/xdocs/language_manual/cli.xml,
but that is splitting hairs :)

"That said, I want to ask:  is this the best use of the development
community's time?  Some of your complaints about the wikidocs are just
glitches in the non-display versions, and presumably those can be fixed or
worked around"

I have made several complains over the past few weeks about the display
versions not looking right, and no one seems to be fixing them. The fact
that the links are broken and have been for some time shows a complete lack
of care from all parties.They should have never been broken in the first
place, if you are making an edit and your not careful enough to test the
page after, that is a bad job, (no offence but sorry). If no one is
watching the edits (I am not) we are not doing a good job either.

"Also, the xdocs weren't removed deliberately -- they broke in December (
HIVE-3896 ) and no one had
the time or inclination to fix them.  Has that situation changed? "
So the xdocs are broke, the wiki is broke, notice a disturbing trend here?
:)

Think about what new users thing when they find out language manual. If I
was a new user and I looked at the wiki, I would just assume that hive was
done by a bunch of cowboy coders, and I probably would not even bother
using it because if someone's main documentation has that many broken links
the software is probably just as bad.

When we originally did the xdoc thing, no one gave a very solid reason as
to why even though reviewing and submitting a patch which usually takes 2
weeks of man hours, and unit tests that take 15 hours to run, that spending
20 minutes writing xdocs was a "great burden".

I think the situation is very different now, as I mentioned we have about
100% turnover in active comitters, Of the active committers hive/hcatalog
there is myself and Alan Gates (think pig book, hive book) we can easily
vote you on as a committer because you have shown a dedication to help with
the documentation situations (committers do not have to be coders).

The argument that killed the xdocs before was "wiki is the status quo".
That argument has clearly fallen apart. We need a better system then
'optimistically hoping that someone maintains the wiki'.



On Mon, Sep 2, 2013 at 5:06 AM, Lefty Leverenz wrote:

> Last year I converted the entire Language Manual to xdocs, but the project
> died from lack of community interest.  So if this goes forward, please
> don't start from scratch again -- my files would need to be updated, but
> wiki page history makes that fairly easy.  Of course there are several new
> docs which would have to be converted from wiki markup to xdocs.
>
> That said, I want to ask:  is this the best use of the development
> community's time?  Some of your complaints about the wikidocs are just
> glitches in the non-display versions, and presumably those can be fixed or
> worked around.  Adding and improving the content seems crucial, but why not
> do that in the wiki?
>
> Also, the xdocs weren't removed deliberately -- they broke in December (
> HIVE-3896 ) and no one
> had the time or inclination to fix them.  Has that situation changed?
>
> -- Lefty Leverenz, devil's advocate
>
>
>
> On Sun, Sep 1, 2013 at 5:47 PM, Edward Capriolo wrote:
>
>> I am not sure about BNF. Hive uses antlr so the language itself is never
>> described as BNF. Maybe antlr has a tool or clever way to turn the .g file
>> into BNF. If it is possible that should be something we do during a
>> document generating step. Also if a new feature does change the language
>> the theory would be the feature would not be committed unless it had
>> associated documentation.
>>
>>
>>
>> On Sun, Sep 1, 2013 at 5:44 PM, Stephen Boesch  wrote:
>>
>>> Will this allow BNF's for the DDL / DML to be  provided and made up to
>>> date  more readily ?
>>>
>>>
>>> 2013/9/1 Edward Capriolo 
>>>
 Over the past few weeks I have taken several looks over documents in
 our wiki.
 The page that strikes me as alarmingly poor is the:
 https://cwiki.apache.org/Hive/languagemanual.html

 This page has several critical broken links such as
 https://cwiki.apache.org/Hive/languagemanual-groupby.html
 https://cwiki.apache.org/Hive/languagemanual-transform.html

 The language manual used to be in decent shape. At times it had
 omissions or was not clear about what version something appeared it, but it
 was very usable.

 A long time ago I had began and completed moving the wiki documentation
 inside the project as xdoc. After completion, several had a problem with
 the xdocs approach. The main complaint was the x

Re: DISCUSS: Hive language manual to be source control managed

[Dean Wampler]

I suggest moving it to GitHub Pages where you use Markdown.

http://pages.github.com/


On Sun, Sep 1, 2013 at 4:39 PM, Edward Capriolo wrote:

> Over the past few weeks I have taken several looks over documents in our
> wiki.
> The page that strikes me as alarmingly poor is the:
> https://cwiki.apache.org/Hive/languagemanual.html
>
> This page has several critical broken links such as
> https://cwiki.apache.org/Hive/languagemanual-groupby.html
> https://cwiki.apache.org/Hive/languagemanual-transform.html
>
> The language manual used to be in decent shape. At times it had omissions
> or was not clear about what version something appeared it, but it was very
> usable.
>
> A long time ago I had began and completed moving the wiki documentation
> inside the project as xdoc. After completion, several had a problem with
> the xdocs approach. The main complaint was the xdoc approach was too
> cumbersome. (However we have basically had a 'turn over' and since that
> time I am one of the few active committers)
>
> The language manual is in very poor shape at the moment with broken links,
> incorrect content, incomplete content, and poor coverage of the actual
> languages. IMHO the attempts to crowd-source this documentation has failed.
> Having a good concise language manual is critical to the success and
> adoption of hive.
>
> I do not believe all of our documentation needs to be in xdoc (as in every
> udf, or every input format) but I believe the language manual surely does.
>
> Please review the current wiki and discuss the concept of moving the
> language manual to source control, or suggest other options.
>
> Thank you,
> Edward
>
>
>


-- 
Dean Wampler, Ph.D.
@deanwampler
http://polyglotprogramming.com

Re: Hive JDBC Server: java.lang.IllegalStateException: Shutdown in progress

[Nitin Pawar]

Can you share what version of hadoop and hive are you using?

This looks similar to HDFS-4841


On Mon, Sep 2, 2013 at 4:20 PM, Guy Doulberg wrote:

> Hi guys,
>
> I have a hive JDBC server in production,
> It started lately to fail.
>
> In the log files I can see the following:
>
> 2013-09-02_10:42:53.13215 java.lang.**IllegalStateException: Shutdown in
> progress, cannot add a shutdownHook
> 2013-09-02_10:42:53.13215   at org.apache.hadoop.util.**
> ShutdownHookManager.**addShutdownHook(**ShutdownHookManager.java:152)
> 2013-09-02_10:42:53.13216   at org.apache.hadoop.fs.**
> FileSystem$Cache.getInternal(**FileSystem.java:2341)
> 2013-09-02_10:42:53.13216   at org.apache.hadoop.fs.**
> FileSystem$Cache.get(**FileSystem.java:2313)
> 2013-09-02_10:42:53.13217   at org.apache.hadoop.fs.**
> FileSystem.get(FileSystem.**java:351)
> 2013-09-02_10:42:53.13217   at org.apache.hadoop.fs.Path.**
> getFileSystem(Path.java:194)
> 2013-09-02_10:42:53.13219   at org.apache.hadoop.hive.ql.**
> exec.Utilities.realFile(**Utilities.java:1027)
> 2013-09-02_10:42:53.13219   at org.apache.hadoop.hive.ql.**
> exec.Utilities.**getResourceFiles(Utilities.**java:1551)
> 2013-09-02_10:42:53.13220   at org.apache.hadoop.hive.ql.**
> exec.ExecDriver.initialize(**ExecDriver.java:152)
> 2013-09-02_10:42:53.13220   at org.apache.hadoop.hive.ql.**
> Driver.launchTask(Driver.java:**1343)
> 2013-09-02_10:42:53.13221   at org.apache.hadoop.hive.ql.**
> Driver.execute(Driver.java:**1138)
> 2013-09-02_10:42:53.13221   at org.apache.hadoop.hive.ql.**
> Driver.run(Driver.java:951)
> 2013-09-02_10:42:53.13222   at org.apache.hadoop.hive.**
> service.HiveServer$**HiveServerHandler.execute(**HiveServer.java:198)
> 2013-09-02_10:42:53.13224   at org.apache.hadoop.hive.**
> service.ThriftHive$Processor$**execute.getResult(ThriftHive.**java:644)
> 2013-09-02_10:42:53.13224   at org.apache.hadoop.hive.**
> service.ThriftHive$Processor$**execute.getResult(ThriftHive.**java:628)
> 2013-09-02_10:42:53.13225   at org.apache.thrift.**
> ProcessFunction.process(**ProcessFunction.java:39)
> 2013-09-02_10:42:53.13225   at org.apache.thrift.**
> TBaseProcessor.process(**TBaseProcessor.java:39)
> 2013-09-02_10:42:53.13226   at org.apache.thrift.server.**
> TThreadPoolServer$**WorkerProcess.run(**TThreadPoolServer.java:206)
> 2013-09-02_10:42:53.13226   at java.util.concurrent.**
> ThreadPoolExecutor$Worker.**runTask(ThreadPoolExecutor.**java:886)
> 2013-09-02_10:42:53.13227   at java.util.concurrent.**
> ThreadPoolExecutor$Worker.run(**ThreadPoolExecutor.java:908)
> 2013-09-02_10:42:53.13227   at java.lang.Thread.run(Thread.**java:662)
> 2013-09-02_10:42:53.13228
> 2013-09-02_10:42:53.13761 FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.**exec.MapRedTask
> 2013-09-02_10:42:53.13763 FAILED: Execution Error, return code 3 from
> org.apache.hadoop.hive.ql.**exec.MapRedTask
>
>
> Has some one encoutered this problem, and know why it is happening?
>
> The jmx doesn't expose anything interesting.
>
>
> Guy
>
>


-- 
Nitin Pawar

Re: Hive Query - Issue

[manish dunani]

Hello,

I think you are working with dynamic partition.
Then you do not need to mention it's value.you only need to put partition
like this:::

try this::

insert overwrite table table_baseline partition (sourcedate) select * from
(select * from table_a where sourcedate='tablea_2013_08' union all select *
from table_b where sourcedate='tableb_2013_08') final

If your problem will not resolve from this then you need to count the no.of
columns in table i think problem is there then correct it if you not and
try once again.I also found the same error in past..



On Mon, Sep 2, 2013 at 5:02 PM, Manickam P  wrote:

>  Hello Experts,
>
> when i try to execute the below query i'm getting error. Please help me to
> correct this.
>
> insert overwrite table table_baseline partition
> (sourcedate='base_2013_08') select * from (select * from table_a where
> sourcedate='tablea_2013_08' union all select * from table_b where
> sourcedate='tableb_2013_08') final
>
> My intention here is i want to populate the table_baseline by using the
> all records from table_a and table_b with partition. I am getting the below
> error.
>
> *Error in semantic analysis: Line 1:23 Cannot insert into target table
> because column number/types are different ''BASE_2013_08'': Table
> insclause-0 has 62 columns, but query has 63 columns.*
>
> I verified the column count and types everything is same but here it says
> some difference. The same query works fine without having any partitions in
> all the three tables but getting error while executing with partitions.
>
>
> please help.
>
>
>
> Thanks
> Manickam P
>



-- 
Regards

*Manish Dunani*
*Contact No* : +91 9408329137
*skype id* : manish.dunani*
*

Hive Query - Issue

[Manickam P]

Hello Experts, 
when i try to execute the below query i'm getting error. Please help me to 
correct this.
insert overwrite table table_baseline partition (sourcedate='base_2013_08') 
select * from (select * from table_a where sourcedate='tablea_2013_08' union 
all select * from table_b where sourcedate='tableb_2013_08') final
My intention here is i want to populate the table_baseline by using the all 
records from table_a and table_b with partition. I am getting the below error. 
Error in semantic analysis: Line 1:23 Cannot insert into target table because 
column number/types are different ''BASE_2013_08'': Table insclause-0 has 62 
columns, but query has 63 columns.
I verified the column count and types everything is same but here it says some 
difference. The same query works fine without having any partitions in all the 
three tables but getting error while executing with partitions. 

please help.


ThanksManickam P
  

Hive JDBC Server: java.lang.IllegalStateException: Shutdown in progress

[Guy Doulberg]

Hi guys,

I have a hive JDBC server in production,
It started lately to fail.

In the log files I can see the following:

2013-09-02_10:42:53.13215 java.lang.IllegalStateException: Shutdown in 
progress, cannot add a shutdownHook
2013-09-02_10:42:53.13215   at 
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
2013-09-02_10:42:53.13216   at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2341)
2013-09-02_10:42:53.13216   at 
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2313)
2013-09-02_10:42:53.13217   at 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
2013-09-02_10:42:53.13217   at 
org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
2013-09-02_10:42:53.13219   at 
org.apache.hadoop.hive.ql.exec.Utilities.realFile(Utilities.java:1027)
2013-09-02_10:42:53.13219   at 
org.apache.hadoop.hive.ql.exec.Utilities.getResourceFiles(Utilities.java:1551)
2013-09-02_10:42:53.13220   at 
org.apache.hadoop.hive.ql.exec.ExecDriver.initialize(ExecDriver.java:152)
2013-09-02_10:42:53.13220   at 
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1343)
2013-09-02_10:42:53.13221   at 
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
2013-09-02_10:42:53.13221   at 
org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
2013-09-02_10:42:53.13222   at 
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
2013-09-02_10:42:53.13224   at 
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
2013-09-02_10:42:53.13224   at 
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
2013-09-02_10:42:53.13225   at 
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
2013-09-02_10:42:53.13225   at 
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2013-09-02_10:42:53.13226   at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
2013-09-02_10:42:53.13226   at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
2013-09-02_10:42:53.13227   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

2013-09-02_10:42:53.13227   at java.lang.Thread.run(Thread.java:662)
2013-09-02_10:42:53.13228
2013-09-02_10:42:53.13761 FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-09-02_10:42:53.13763 FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.MapRedTask



Has some one encoutered this problem, and know why it is happening?

The jmx doesn't expose anything interesting.


Guy

Re: DISCUSS: Hive language manual to be source control managed

[Lefty Leverenz]

Last year I converted the entire Language Manual to xdocs, but the project
died from lack of community interest.  So if this goes forward, please
don't start from scratch again -- my files would need to be updated, but
wiki page history makes that fairly easy.  Of course there are several new
docs which would have to be converted from wiki markup to xdocs.

That said, I want to ask:  is this the best use of the development
community's time?  Some of your complaints about the wikidocs are just
glitches in the non-display versions, and presumably those can be fixed or
worked around.  Adding and improving the content seems crucial, but why not
do that in the wiki?

Also, the xdocs weren't removed deliberately -- they broke in December (
HIVE-3896 ) and no one had
the time or inclination to fix them.  Has that situation changed?

-- Lefty Leverenz, devil's advocate



On Sun, Sep 1, 2013 at 5:47 PM, Edward Capriolo wrote:

> I am not sure about BNF. Hive uses antlr so the language itself is never
> described as BNF. Maybe antlr has a tool or clever way to turn the .g file
> into BNF. If it is possible that should be something we do during a
> document generating step. Also if a new feature does change the language
> the theory would be the feature would not be committed unless it had
> associated documentation.
>
>
>
> On Sun, Sep 1, 2013 at 5:44 PM, Stephen Boesch  wrote:
>
>> Will this allow BNF's for the DDL / DML to be  provided and made up to
>> date  more readily ?
>>
>>
>> 2013/9/1 Edward Capriolo 
>>
>>> Over the past few weeks I have taken several looks over documents in our
>>> wiki.
>>> The page that strikes me as alarmingly poor is the:
>>> https://cwiki.apache.org/Hive/languagemanual.html
>>>
>>> This page has several critical broken links such as
>>> https://cwiki.apache.org/Hive/languagemanual-groupby.html
>>> https://cwiki.apache.org/Hive/languagemanual-transform.html
>>>
>>> The language manual used to be in decent shape. At times it had
>>> omissions or was not clear about what version something appeared it, but it
>>> was very usable.
>>>
>>> A long time ago I had began and completed moving the wiki documentation
>>> inside the project as xdoc. After completion, several had a problem with
>>> the xdocs approach. The main complaint was the xdoc approach was too
>>> cumbersome. (However we have basically had a 'turn over' and since that
>>> time I am one of the few active committers)
>>>
>>> The language manual is in very poor shape at the moment with broken
>>> links, incorrect content, incomplete content, and poor coverage of the
>>> actual languages. IMHO the attempts to crowd-source this documentation has
>>> failed. Having a good concise language manual is critical to the success
>>> and adoption of hive.
>>>
>>> I do not believe all of our documentation needs to be in xdoc (as in
>>> every udf, or every input format) but I believe the language manual surely
>>> does.
>>>
>>> Please review the current wiki and discuss the concept of moving the
>>> language manual to source control, or suggest other options.
>>>
>>> Thank you,
>>> Edward
>>>
>>>
>>>
>>
>

Re: What is this windowing query supposed to do?

[Lefty Leverenz]

Once again the /confluence/display version of the wiki is fine while the
non-display version is glitchy.  First the {code} tags gave invisible
sample code, and now some of the doc links don't work -- but they're fine
here:  https://cwiki.apache.org/confluence/display/Hive/LanguageManual.

Can anyone explain this, or better yet solve it?  Until it's solved, should
we include a link to the display version at the top of each wikidoc?  (And
I suppose that's the royal "we" unless someone else volunteers.)

For the windowing
doc,
I've added a link to Harish's PDF file in the first paragraph:

   - See "Windowing Specifications in
HQL"
   (attached to HIVE-4197 )
   for details.

Happy Labor Day, everyone.  -- Lefty



On Sun, Sep 1, 2013 at 8:36 PM, Harish Butani wrote:

> In your 3rd query the OVER clause is being ignored because there is no
> associated Aggregation Function.
> This has been fixed in HIVE-4214. The query fails with the following error:
> 'cannot recognize input near 'range' 'OVER' '(' in select expression'
>
> The Windowing behavior in Hive is documented in HIVE-4197
>
> https://issues.apache.org/jira/secure/attachment/12575830/WindowingSpecification.pdf
>
> On Sep 1, 2013, at 2:04 PM, Edward Capriolo  wrote:
>
> hive> select * from field_goal;
> OK
> tynes44miss
> tynes40miss
> tynes33make
> pete20make
> pete21make
> arnold56make
> arnold22make
> arnold45miss
> Time taken: 0.04 seconds, Fetched: 8 row(s)
>
> This makes sense:
> hive> select name,avg(range) OVER (PARTITION by name ORDER by range) FROM
> field_goal  ;
> arnold22.0
> arnold33.5
> arnold41.0
> pete20.0
> pete20.5
> tynes33.0
> tynes36.5
> tynes39.0
> Time taken: 4.14 seconds, Fetched: 8 row(s)
>
> but what is this?
> hive> select name,range OVER (PARTITION by name ORDER by range) FROM
> field_goal  ;
> tynes44
> tynes40
> tynes33
> pete20
> pete21
> arnold56
> arnold22
> arnold45
> Time taken: 3.709 seconds, Fetched: 8 row(s)
>
> It would be great if the
> https://cwiki.apache.org/Hive/languagemanual-windowingandanalytics.html#LanguageManualWindowingAndAnalytics-PARTITIONBYwithpartitioning%252CORDERBY%252Candwindowspecification
>
>
> Most of the links here:
> https://cwiki.apache.org/Hive/languagemanual.html
>
> are broken...
> What gives?
> The language manual used to be fairly useful, now it is in major dis-array.
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.