Re: HDP, Hive + Ignite

Hi Ivan, TEZ was on 6 data nodes. So you're right, I can't reliably estimate the performance of Ignite MR. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12868.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

TEZ (80 sec vs 23 sec). I guess because of one ignite node. On > multi node cluster result was incorrect. > > 2. "Do I correctly understand that Ignite MR was not used in that > experiment?" > Yes, it was TEZ+IGFS. > > > > -- > View this message in contex

Re: HDP, Hive + Ignite

it was TEZ+IGFS. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12850.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

han Hive on TEZ, but faster than Hive on MR. 2) TEZ+HDFS and TEZ+IGFS work at the same speed. Although TEZ+IGFS can be faster in queries with intensive I/O (need to test). Many thanks for your patience and prompt help. I'm going to try Ignite + Spark, I'll open new topic) -- View thi

Re: HDP, Hive + Ignite

) TEZ+HDFS and TEZ+IGFS work at the same speed. Although TEZ+IGFS can be >> faster in queries with intensive I/O (need to test). >> >> Many thanks for your patience and prompt help. >> I'm going to try Ignite + Spark, I'll open new topic) >> >> >> >> -- >> View this message in context: http://apache-ignite-users.705 >> 18.x6.nabble.com/HDP-Hive-Ignite-tp12195p12838.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >

Re: HDP, Hive + Ignite

d TEZ+IGFS work at the same speed. Although TEZ+IGFS can be > faster in queries with intensive I/O (need to test). > > Many thanks for your patience and prompt help. > I'm going to try Ignite + Spark, I'll open new topic) > > > > -- > View this message in context: h

Re: HDP, Hive + Ignite

nabble.com/HDP-Hive-Ignite-tp12195p12838.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

process was killed by the operating system. Can you please see the > kernel log -- what does it say near the node crash time? > > > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12660.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >

Re: HDP, Hive + Ignite

.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12660.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

execution time on TEZ (table in HDFS) and TEZ (table in IGFS) is comparable: TEZ: 215 sec (6 nodes) TEZ+IGFS: 207 sec (6 nodes) I'm waiting for the results of your tests. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12638.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

As a workaround to IGNITE-4862 propetry FileSystemConfiguration#perNodeParallelBatchCount can be set to 1. Also setting FileSystemConfiguration#prefetchBlocks to 0 should help. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12627.html

Re: HDP, Hive + Ignite

I keep in mind ticket for TEZ experimenting\investigations. Evgeniy, sure, IGNITE-4862, the link is above. On Thu, May 11, 2017 at 11:07 AM, Evgeniy Stanilovskiy wrote: Ivan, do we have appropriate jira ticket? 4. No. We only start experimenting with Tez -- I'm

Re: HDP, Hive + Ignite

Alena, regarding NPEs in Ignite node logs, this seems to be https://issues.apache.org/jira/browse/IGNITE-4862 , fixed, but not yet merged. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12594.html Sent from the Apache Ignite Users

Re: HDP, Hive + Ignite

ults of your experiments. p.s. I sent email again on 5th May. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12588.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

.com/file/ > n12457/ignite-node-dn2.log> > > 4. I think Evgeniy said about comparison TEZ vs TEZ+IGFS. So I join the > question: > did you conduct some tests Ignite + TEZ? > > p.s. Ivan, did you get my email about Hadoop meetup? I sent it couple of > days ago. >

Re: HDP, Hive + Ignite

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12457.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

ext: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12430.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

ng IGFS here you simply append additional memory burden here. TEZ standalone is good enough. Ivan, does Ignite have some tests in case of TEZ usage ? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12419.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

napshot" line in the log. It should say "Topology snapshot [ver=1, servers=1, clients=0, CPUs=..." . If topology version is different from 1, that means something is wrong, possibly there is a stale Ignite process this one attempts to join. -- View this message in context: http://a

Re: HDP, Hive + Ignite

this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12398.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

, heap=10.0GB] 4. Can you please specify more exactly, what Evgeniy's comment you're referring to? Regards, Ivan Veselovsky. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12394.html Sent from the Apache Ignite Users mailing list archive

Re: HDP, Hive + Ignite

data from the disk. Still hope this is possible) -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12393.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

.nabble.com/HDP-Hive-Ignite-tp12195p12355.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

gnite-node-dn2.log> 3. When I start ignite nodes on different servers sometimes they do not see each other. I have to rerun a node a few times, after that they connect in one cluster. Is it normal? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Igni

Re: HDP, Hive + Ignite

Yes, we did some experiments with Hive over Ignite on HDP distributions, in basic experiments everything was working without critical issues. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12308.html Sent from the Apache Ignite Users

Re: HDP, Hive + Ignite

operty is *required* to execute Hive queries on Ignite MR successfully. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12307.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

ubts that this is possible https://community.hortonworks.com/questions/24953/solution-for-hive-runtime-error-while-processing-r.html -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12304.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

by the Ignite java process -- you can use a native (your OS specific) process monitoring tool for that . -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12296.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

ow Ignite will utilize memory for that? > 1 Ignite node per host is okay, if the node can utilize all the host > resources (memory & cpu) How can I find out that ignite node utilizes all the host resources? Thanks for your help! -- View this message in context: http://apache-

Re: HDP, Hive + Ignite

ode per host is okay, if the node can utilize all the host resources (memory & cpu). 7. Yes. Ignite does not give MR task execution correctness guarantees if a node is stopped or crashed. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12248.h

Re: HDP, Hive + Ignite

execution then job (query execution) failed. Is it normal? Thanks in advance! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/HDP-Hive-Ignite-tp12195p12238.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: HDP, Hive + Ignite

p.s. Please use a HadoopFileSystemFactory in secondary file system config, as described there https://apacheignite-fs.readme.io/docs/installing-on-hortonworks-hdp , constructor org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem(2) is deprecated. On Mon, Apr 24, 2017 at 7:44 PM, Ivan

Re: HDP, Hive + Ignite

Hi, Aloha, First of all, Hadoop Accelerator consists of 2 parts that are independent and can be used one without the other: (1) IGFS and (2) map-reduce execution engine. IGFS is not used in your case because default file system in your cluster is still hdfs:// (specified by global property