Re: Using Spark on Hive with Hive also using Spark as its execution engine

> That being said all systems are evolving. Hive supports tez+llap which >is basically the in-memory support. There is a big difference between where LLAP & SparkSQL, which has to do with access pattern needs. The first one is related to the lifetime of the cache - the Spark RDD cache is per-use

Re: Using Spark on Hive with Hive also using Spark as its execution engine

Mich - it sounds like maybe you should try these benchmarks with alluxio abstracting the storage layer, and see how much it makes a difference. Alluxio should (if I understand it right) provide a lot of the optimisation you're looking for with in memory work. I've never used it, but I would love t

Re: Using Spark on Hive with Hive also using Spark as its execution engine

I think we are going to move to a model that the computation stack will be separate from storage stack and moreover something like Hive that provides the means for persistent storage (well HDFS is the one that stores all the data) will have an in-memory type capability much like what Oracle TimesTe

Re: Using Spark on Hive with Hive also using Spark as its execution engine

And you have MapR supporting Apache Drill. So these are all alternatives to Spark, and its not necessarily an either or scenario. You can have both. > On May 30, 2016, at 12:49 PM, Mich Talebzadeh > wrote: > > yep Hortonworks supports Tez for one reason or other which I am going > hopefull

Re: Using Spark on Hive with Hive also using Spark as its execution engine

I do not think that in-memory itself will make things faster in all cases. Especially if you use Tez with Orc or parquet. Especially for ad hoc queries on large dataset (indecently if they fit in-memory or not) this will have a significant impact. This is an experience I have also with the in-m

Re: Using Spark on Hive with Hive also using Spark as its execution engine

yep Hortonworks supports Tez for one reason or other which I am going hopefully to test it as the query engine for hive. Tthough I think Spark will be faster because of its in-memory support. Also if you are independent then you better off dealing with Spark and Hive without the need to support an

Re: Using Spark on Hive with Hive also using Spark as its execution engine

Mich, Most people use vendor releases because they need to have the support. Hortonworks is the vendor who has the most skin in the game when it comes to Tez. If memory serves, Tez isn’t going to be M/R but a local execution engine? Then LLAP is the in-memory piece to speed up Tez? HTH -M

Re: SHOW DATABASES/TABLES with SQL standard authorization

ok that is different from seeing the list of databases. That is just schema case in point in SAP ASE a normal RDBMS > sp_addlogin someuser, someuser123, scratchpad 2> go Password correctly set. Account unlocked. New login created. (return status = 0) 1> exit isql -U someuser -w1000 Password: --

Is it possible to use external table on top of Elasticsearch index for arbitrary FTS

I know that external table can be defined like this CREATE EXTERNAL TABLE artists ( id BIGINT, nameSTRING, links STRUCT) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'radio/artists', 'es.query' = '?q=me*'); But can I just define

Re: SHOW DATABASES/TABLES with SQL standard authorization

I realize it's just a list, but it's still not something I'd expect. The difference compared to a normal RDBMS is that they typically have a CONNECT privilege, which I can use to restrict the user to connecting to other databases. I'm also more concerned about SHOW TABLES. It just seems strange th

Re: SHOW DATABASES/TABLES with SQL standard authorization

the behaviour is no different from a normal RDBMS. show databases actually inquires Hive metadata table DBS. select NAME, OWNER_NAME from DBS order by 1,2; NAME OWNER_NAME -- -- accounts hdus

SHOW DATABASES/TABLES with SQL standard authorization

I'm setting up a Hive instance with the SQL standard authorization and it seems to be working great for all normal operations, but for SHOW DATABASES/TABLES it's behaving differently from what I would expect. It always shows all databases/tables, even though I do not have access to those tables. I

RE: Does hive need exact schema in Hive Export/Import?

Hi 1) I was able to do the import by doing the following manipulation: · Export table dev101 · Create an empty table dev102 · Export table dev102 · replace the _metadata file of dev101 with the _metadata file of dev102 · import table dev101 to table de

Re: Permission denied with select *

Access control lists... Who owns the system, database, tables, & files? Not only owner, but also do you belong to the group? What are the permissions on the files behind the table? Is Sentry enabled? On May 30, 2016 9:52 AM, "kishore kumar" wrote: Hi, If i run "select * from table" permission

Permission denied with select *

Hi, If i run "select * from table" permission denied error we are encountering, where as select * from table limit 10; or select count(*) from table; working fine, what could be the reason any guess ? -- Thanks, Kishore.

Re: Does hive need exact schema in Hive Export/Import?

I guess one alternative is to import it AS IS (the same column type) to a staging table and then do insert/select into the target table from the staging table. import/export is for coping data from say prod to dev like to like. the problem is that it does two things. it exports both data and meta

Re: My first TEZ job fails

> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar >./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt >/tmp/out/test.log Sure, you're missing file:/// - the defaultFS is most like hdfs://:/ The inputs and outputs without a scheme prefix will go the defaultFS configured in cor

Re: Does hive need exact schema in Hive Export/Import?

Hi Mich, you did not get my question I guess . I am able to use import export. I am exporting data from dev101 (col1 int, col2 char(10)) and importing in dev102 (col1 int, col2 string) I am getting issue : >FAILED: SemanticException [Error 10120]: The existing table is not compatible >

Re: Anyone successfully deployed Hive on TEZ engine?

Hi Gopal, please see my correspondence about Tez in tez user group. I forwarded to hive user group. thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Does hive need exact schema in Hive Export/Import?

it is pretty straight forward !hdfs dfs -rm -r hdfs://rhes564:9000/export; EXPORT TABLE oraclehadoop.sales_staging to "hdfs://rhes564:9000/export"; -- DROP TABLE IF EXISTS test.sales_staging; IMPORT TABLE test.sales_staging FROM "hdfs://rhes564:9000/export"; select count(1) from test.sales_stagi

Does hive need exact schema in Hive Export/Import?

Hi All, I am using HDP 2.3 - Hadoop version - 2.7.1 - Hive version - 1.2.1 I created a table dev101 in hive using create table dev101 (col1 int, col2 char(10)); I inserted two records using insert into dev101 values (1, 'value1'); insert into dev101 values (2, 'value2'); I exp

Re: Anyone successfully deployed Hive on TEZ engine?

> I do not use any vendor's product., All my own set up, build and >configure. My autobuild scripts should serve as readable documentation for this, since nearly everything's in a single Makefile with an install: target. Or take the easy route with $ make dist install In case you use the llap b

Re: Anyone successfully deployed Hive on TEZ engine?

Thanks Gopal. I do not use any vendor's product., All my own set up, build and configure. No CDH no HDL etc. This the current stack that I have: Java *java -version* java version "1.8.0_77" Java(TM) SE Runtime Environment (build 1.8.0_77-b03) Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03,

Re: Anyone successfully deployed Hive on TEZ engine?

> In short at the simplest set up what Resource Manager it works with? Tez+Hive needs HDFS and YARN 2.6.0+ (preferably as close to an Apache build as possible - CDH clusters need more work). Hive2 needs Apache Slider 0.91 right now, to start the cache daemons on YARN (see SLIDER-82). > If so ki

Re: Anyone successfully deployed Hive on TEZ engine?

thanks Damien. I tried TEZ 0.82 with Hive 2 although I did not persevere. When you say "Not stable" are you referring to using it with YARN etc. In short at the simplest set up what Resource Manager it works with? Cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?i

Re: Anyone successfully deployed Hive on TEZ engine?

HIVE 1.2.1 and Tez 0.5.2 or 0.7.0 works pretty well beginning to use HIVE 2.0.0 and 0.8.x but not stable :/ 2016-05-29 22:26 GMT+02:00 Mich Talebzadeh : > > Please bear in mind that I am talking about your own build not anything > comes as part of Vendor's package. > > If so kindly specify both