Re: LLAP woes...
Hello Hive Users, Does the version of zookeeper matter with these decode errors or is this more of a jar issue? I can do a get in zkCli for these paths and do get data for the LLAP instances. hive 3.1.2 tried zookeeper: 3.4.12, 3.4.16, and 3.4.6. This error is when running 3.4.6. 2019-09-04T19:09:00,190 INFO [hiveserver2-web-42] utils.ServiceApiUtil: Loading service definition from hdfs://ip-10-199-1-216.ec2.internal:8020/user/hadoop/.yarn/services/looker-llap/looker-llap.json 2019-09-04T19:09:00,595 INFO [hiveserver2-web-42] impl.LlapRegistryService: Using LLAP registry (client) type: Service LlapRegistryService in state LlapRegistryService: STARTED 2019-09-04T19:09:00,746 ERROR [hiveserver2-web-42] impl.ZkRegistryBase: Unable to decode data for zkpath: /user-hadoop/looker-llap/workers/worker-04. Ignoring from current instances list.. 2019-09-04T19:09:00,746 ERROR [hiveserver2-web-42] impl.ZkRegistryBase: Unable to decode data for zkpath: /user-hadoop/looker-llap/workers/worker-06. Ignoring from current instances list.. On Wed, Sep 4, 2019 at 1:11 PM dan young wrote: > Hello Hive, > > We're trying to get LLAP running on Hive 3.1.2. We're having some issues > and I'm trying to figure out why the Hive Web UI is reporting a status of > LAUNCHING although it seems that the LLAP daemons are running. I can see > the LlapDaemon running, no errors the the llap container logs, nothing > jumps out, etc One thing I do noticed is that when we click on the Hive > web UI, we get an Unable to decode data path for zkpath... error in the > HiveServer2 log > > 2019-09-04T19:09:00,190 INFO [hiveserver2-web-42] utils.ServiceApiUtil: > Loading service definition from > hdfs://ip-10-199-1-216.ec2.internal:8020/user/hadoop/.yarn/services/looker-llap/looker-llap.json > > 2019-09-04T19:09:00,595 INFO [hiveserver2-web-42] > impl.LlapRegistryService: Using LLAP registry (client) type: Service > LlapRegistryService in state LlapRegistryService: STARTED > > 2019-09-04T19:09:00,746 ERROR [hiveserver2-web-42] impl.ZkRegistryBase: > Unable to decode data for zkpath: > /user-hadoop/looker-llap/workers/worker-04. Ignoring from current > instances list.. > > 2019-09-04T19:09:00,746 ERROR [hiveserver2-web-42] impl.ZkRegistryBase: > Unable to decode data for zkpath: > /user-hadoop/looker-llap/workers/worker-06. Ignoring from current > instances list.. > > > does anyone have some debugging tips and tricks? or insight into what to > look at next? > >
Re: May 2018 Hive User Group Meeting
Looks like great talks, will this be streamed anywhere? On Wed, May 2, 2018, 8:48 AM Sahil Takiar wrote: > Hey Everyone, > > The agenda for the meetup has been set and I'm excited to say we have lots > of interesting talks scheduled! Below is final agenda, the full list of > abstracts will be sent out soon. If you are planning to attend, please RSVP > on the meetup link so we can get an accurate headcount of attendees ( > https://www.meetup.com/Hive-User-Group-Meeting/events/249641278/). > > 6:30 - 7:00 PM Networking and Refreshments > 7:00PM - 8:20 PM Lightning Talks (10 min each) - 8 talks total > >- What's new in Hive 3.0.0 - Ashutosh Chauhan >- Hive-on-Spark at Uber: Efficiency & Scale - Xuefu Zhang >- Hive-on-S3 Performance: Past, Present, and Future - Sahil Takiar >- Dali: Data Access Layer at LinkedIn - Adwait Tumbde >- Parquet Vectorization in Hive - Vihang Karajgaonkar >- ORC Column Level Encryption - Owen O’Malley >- Running Hive at Scale @ Lyft - Sharanya Santhanam, Rohit Menon >- Materialized Views in Hive - Jesus Camacho Rodriguez > > 8:30 PM - 9:00 PM Hive Metastore Panel > >- Moderator: Vihang Karajgaonkar >- Participants: > - Daniel Dai - Hive Metastore Caching > - Alan Gates - Hive Metastore Separation > - Rituparna Agrawal - Customer Use Cases & Pain Points of (Big) > Metadata > > The Metastore panel will consist of a short presentation by each panelist > followed by a Q&A session driven by the moderator. > > On Tue, Apr 24, 2018 at 2:53 PM, Sahil Takiar > wrote: > >> We still have a few slots open for lightening talks, so if anyone is >> interested in giving a presentation don't hesitate to reach out! >> >> If you are planning to attend the meetup, please RSVP on the Meetup link ( >> https://www.meetup.com/Hive-User-Group-Meeting/events/249641278/) so >> that we can get an accurate headcount for food. >> >> Thanks! >> >> --Sahil >> >> On Wed, Apr 11, 2018 at 5:08 PM, Sahil Takiar >> wrote: >> >>> Hi all, >>> >>> I'm happy to announce that the Hive community is organizing a Hive user >>> group meeting in the Bay Area next month. The details can be found at >>> https://www.meetup.com/Hive-User-Group-Meeting/events/249641278/ >>> >>> The format of this meetup will be slightly different from previous ones. >>> There will be one hour dedicated to lightning talks, followed by a group >>> discussion on the future of the Hive Metastore. >>> >>> We are inviting talk proposals from Hive users as well as developers at >>> this time. Please contact either myself (takiar.sa...@gmail.com), >>> Vihang Karajgaonkar (vih...@cloudera.com), or Peter Vary ( >>> pv...@cloudera.com) with proposals. We currently have 5 openings. >>> >>> Please let me know if you have any questions or suggestions. >>> >>> Thanks, >>> Sahil >>> >> >> >> >> -- >> Sahil Takiar >> Software Engineer >> takiar.sa...@gmail.com | (510) 673-0309 >> > > > > -- > Sahil Takiar > Software Engineer > takiar.sa...@gmail.com | (510) 673-0309 >
Re: Hive LLAP service is not starting
Hello Rajesh, Here's what we use on Hive 2.3 and Hadoop 2.7.4 export PATH=$PATH:/usr/lib/slider/bin export HADOOP_CONF_DIR=/usr/lib/hadoop/conf export JAVA_HOME=/usr/java/default export HADOOP_HOME=/usr/lib/hadoop /usr/lib/hive/bin/hive --service llap --name llaptest --instances 2 --size 6144m --cache 512m --slider-am-container-mb 2048 --queue default --executors 4 --xmx 5120m --loglevel INFO --auxjars --args " -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:InitiatingHeapOccupancyPercent=40 -XX:G1ReservePercent=20 -XX:MaxGCPauseMillis=200 -XX:MetaspaceSize=512m" --slider-appconfig-global site.global.app_user=hadoop this will create a run.sh file in the folder that you are running the command from. The run the run.sh, which will submit to YARN. ./llap-slider-24Aug2017/run.sh you can check the status either thru the YARN UI and/or slider command Regards, Dano On Tue, Sep 12, 2017 at 3:15 AM Rajesh Narayanan < rajesh.r.naraya...@oracle.com> wrote: > Hi Premal > > Thanks for the reply. I am following below document. Can you please > provide any document or steps to start LLAP through slide > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/intro_new_book.html > > Regards > > Rajesh > > *From:* Premal Shah [mailto:premal.j.s...@gmail.com] > *Sent:* Monday, September 11, 2017 11:29 PM > *To:* user@hive.apache.org > *Subject:* Re: Hive LLAP service is not starting > > > > Hi Rajesh, > > I thought the recommended way to start/stop LLAP was to use Slider > > this is what we do to start > > > > slider create llap --resources $BASEDIR/resources.json --template > $BASEDIR/appConfig.json > > > > and to stop before you can start again > > > > slider stop llap > > slider destroy llap --force || slider destroy llap > > > > > > On Mon, Sep 11, 2017 at 5:36 AM, Rajesh Narayanan < > rajesh.r.naraya...@oracle.com> wrote: > > Hi All, > > I am using Hadoop 2.7.3 and Hive 2.2.0 and TEZ 0.9.0 in linux. and able to > execute hive query in yarn as TEZ type. > > When I start hive LLAP service -./hive --service llap --name @llap > --instances 1 --cache 1024m --xmx 2048m --size 3225m --loglevel DEBUG > --args " -XX:+UseG1GC -XX:+ResizeTLAB -XX:+UseNUMA -XX:-ResizePLAB" > > Getting below exception. Could you please suggest what missing here. > > > > Failed: java.io.FileNotFoundException: > /tmp/staging-slider-HHIwk3/lib/tez.tar.gz (Is a directory) > java.util.concurrent.ExecutionException: java.io.FileNotFoundException: > /tmp/staging-slider-HHIwk3/lib/tez.tar.gz (Is a directory) > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:605) > at > org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:113) > Caused by: java.io.FileNotFoundException: > /tmp/staging-slider-HHIwk3/lib/tez.tar.gz (Is a directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at > org.apache.hadoop.hive.common.CompressionUtils.unTar(CompressionUtils.java:152) > at > org.apache.hadoop.hive.llap.cli.LlapServiceDriver$1.call(LlapServiceDriver.java:361) > at > org.apache.hadoop.hive.llap.cli.LlapServiceDriver$1.call(LlapServiceDriver.java:348) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > INFO cli.LlapServiceDriver: LLAP service driver finished > > > > Thanks > > Rajesh > > > > > > -- > > Regards, > Premal Shah. >
Re: Aug. 2017 Hive User Group Meeting
Dooh..thanx! On Tue, Aug 22, 2017, 11:11 AM Alan Gates wrote: > The address is at the top of the text description, even though it isn’t in > the location field: > > 5470 Great America Parkway, Santa Clara, CA > > Alan. > > On Mon, Aug 21, 2017 at 5:50 PM, dan young wrote: > >> For us out of town folks, where is the location of this meetup? Says >> Hortonworks but do you have an address? >> >> Regards >> >> Dano >> >> On Mon, Aug 21, 2017, 1:33 PM Xuefu Zhang wrote: >> >>> Dear Hive users and developers, >>> >>> As reminder, the next Hive User Group Meeting will occur this Thursday, >>> Aug. 24. The agenda is available on the event page ( >>> https://www.meetup.com/Hive-User-Group-Meeting/events/242210487/). >>> >>> See you all there! >>> >>> Thanks, >>> Xuefu >>> >>> On Tue, Aug 1, 2017 at 7:18 PM, Xuefu Zhang wrote: >>> >>>> Hi all, >>>> >>>> It's an honor to announce that Hive community is launching a Hive user >>>> group meeting in the bay area this month. The details can be found at >>>> https://www.meetup.com/Hive-User-Group-Meeting/events/242210487/. >>>> >>>> We are inviting talk proposals from Hive users as well as developers at >>>> this time. We currently have 5 openings. >>>> >>>> Please let me know if you have any questions or suggestions. >>>> >>>> Thanks, >>>> Xuefu >>>> >>>> >>> >
Re: Aug. 2017 Hive User Group Meeting
For us out of town folks, where is the location of this meetup? Says Hortonworks but do you have an address? Regards Dano On Mon, Aug 21, 2017, 1:33 PM Xuefu Zhang wrote: > Dear Hive users and developers, > > As reminder, the next Hive User Group Meeting will occur this Thursday, > Aug. 24. The agenda is available on the event page ( > https://www.meetup.com/Hive-User-Group-Meeting/events/242210487/). > > See you all there! > > Thanks, > Xuefu > > On Tue, Aug 1, 2017 at 7:18 PM, Xuefu Zhang wrote: > >> Hi all, >> >> It's an honor to announce that Hive community is launching a Hive user >> group meeting in the bay area this month. The details can be found at >> https://www.meetup.com/Hive-User-Group-Meeting/events/242210487/. >> >> We are inviting talk proposals from Hive users as well as developers at >> this time. We currently have 5 openings. >> >> Please let me know if you have any questions or suggestions. >> >> Thanks, >> Xuefu >> >> >
Re: Long time compiling query/explain.....
Thank you Gopal, I'll try to upgrade to 2.3 and see how that goes. On Mon, Aug 14, 2017 at 4:08 PM Gopal Vijayaraghavan wrote: > > > > Running Hive 2.2 w/ LLAP enabled (tried the same thing in Hive 2.3 w/ > LLAP), queries working but when we submit queries like the following (via > our automated test framework), they just seem to hang with Parsing > CommandOther queries seem to work fine Any idea on what's going on > or how to debug? I have debug log set and all we're seeing in the > HiveServer2 logs are: > > Get a jstack - like 20 jstacks 1s apart. > > I think this be similar to the parenthesis parsing bug fixed in 2.3.0. > > https://issues.apache.org/jira/browse/HIVE-16074 > > Also getting a dummy table schema would help me run explain on my own > builds. > > Cheers, > Gopal > > > >
Long time compiling query/explain.....
Running Hive 2.2 w/ LLAP enabled (tried the same thing in Hive 2.3 w/
LLAP), queries working but when we submit queries like the following (via
our automated test framework), they just seem to hang with Parsing
CommandOther queries seem to work fine Any idea on what's going on
or how to debug? I have debug log set and all we're seeing in the
HiveServer2 logs are:
2017-08-14T17:02:09,919 DEBUG [33a7d77a-688a-4766-b897-dbffef8ba6d3
HiveServer2-Handler-Pool: Thread-148] log.PerfLogger:
2017-08-14T17:02:09,933 DEBUG [33a7d77a-688a-4766-b897-dbffef8ba6d3
HiveServer2-Handler-Pool: Thread-148] parse.ParseDriver: Parsing command:
SELECT CASE WHEN (((TIMESTAMP('2015-11-30 23:30:00.0')) >=
((TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP(CONCAT(ADD_MONTHS(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01 00:00:00.0')
, 'MM'))),11 AS STRING), '-',
LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3)
+ 1 AS STRING), 2, '0'), '-01 00:00:00') AS TIMESTAMP),
-1),SUBSTRING(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))
),11 AS STRING), '-',
LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS
STRING), 2, '0'), '-01 00:00:00') AS TIMESTAMP),11))), 'MM'))),
-3),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP(CONCAT(ADD_MONTHS(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))), 1),SUBSTRING(TIMESTAMP(DATE(
TRUNC(TIMESTAMP('2015-12-01 00:00:00.0'), 'MM'))),11 AS
STRING), '-',
LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01 00:00:00.00
000'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING), 2, '0'),
'-01 00:00:00') AS TIMESTAMP),
-1),SUBSTRING(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))), 1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TI
MESTAMP('2015-12-01 00:00:00.0'), 'MM'))),11 AS STRING), '-',
LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01 00:00:00.0'
), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING), 2, '0'), '-01
00:00:00') AS TIMESTAMP),11))), 'MM'))),11) AND (TIMESTAMP('2015-11-30
23:30:00.0')) <
((TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP
(CONCAT(ADD_MONTHS(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 AS STRING), '-',
LPAD(CAST(((CAST(CEIL(MONT
H(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING),
2, '0'), '-01 00:00:00') AS TIMESTAMP), -1)
,SUBSTRING(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 AS STRING), '-',
LPAD(CAST(((CAST(CEIL(MONTH(TIMEST
AMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRING),
2, '0'), '-01 00:00:00') AS TIMESTAMP),11))), 'MM')
)),
-3),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP(CONCAT(ADD_MONTHS(CAST(CONCAT(CAST(YEAR(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11
AS STRING), '-',
LPAD(CAST(((CAST(CEIL(MONTH(TIMESTAMP(CONCAT(ADD_MONTHS(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),
1),SUBSTRING(TIMESTAMP(DATE(TRUNC(TIMESTAMP('2015-12-01
00:00:00.0'), 'MM'))),11 / 3) AS INT) - 1) * 3) + 1 AS STRI
NG), 2, '0'), '-01 00:00:00') AS TIMESTAMP),
-1),SUBSTRING(CAST(CONCAT(CAST
