Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Hi Mikhail, Your last reply is showing as blank. Any luck with reproduce? Thanks! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725p16217.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Let me try to clear here with the sequence of steps performed. - Created table with partition through hive using below query. It creates a directory in hdfs. create table stocks3 (stock string, time timestamp, price float) PARTITIONED BY (years bigint, months bigint, days bigint) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; - Then I get streaming data and using IgniteFileSystem's append/create method, it gets saved to ignited hadoop. - Run below select query. No result returned select * from stocks3; - Stop ignite and run the select again on hive. No result with below logs hive> select * from stocks3; 17/08/04 14:59:08 INFO conf.HiveConf: Using the default value passed in for log id: b5e3e924-e46a-481c-8aef-30d48605a2da 17/08/04 14:59:08 INFO session.SessionState: Updating thread name to b5e3e924-e46a-481c-8aef-30d48605a2da main 17/08/04 14:59:08 WARN operation.Operation: Unable to create operation log file: D:\tmp\hive\\operation_logs\b5e3e924-e46a-481c-8aef-30d48605a2da\137adad6-ea23-462c-a414-6ce260e5bd49 java.io.IOException: The system cannot find the path specified at java.io.WinNTFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:1012) at org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:237) at org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:279) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:314) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:499) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:486) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1412) at com.sun.proxy.$Proxy21.ExecuteStatement(Unknown Source) at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:308) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250) at org.apache.hive.beeline.Commands.executeInternal(Commands.java:988) at org.apache.hive.beeline.Commands.execute(Commands.java:1160) at org.apache.hive.beeline.Commands.sql(Commands.java:1074) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1148) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:976) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886) at org.apache.hive.beeline.cli.HiveCli.runWithArgs(HiveCli.java:35) at org.apache.hive.beeline.cli.HiveCli.main(HiveCli.java:29) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:491) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) 17/08/04 14:59:08 INFO ql.Driver: Compiling command(queryId=_20170804145908_b270c978-ab00-4160-a2a6-c19b42eab676): select * from stocks3 17/08/04 14:59:08 INFO parse.CalcitePlanner: Starting Semantic Analysis 17/08/04 14:59:08 INFO parse.CalcitePlanner: Completed phase 1 of Semantic Analysis 17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for source tables 17/08/04 14:59:08 INFO metastore.HiveMetaStore: 0: get_table : db=yt tbl=stocks3 17/08/04 14:59:08 INFO HiveMetaStore.audit: ugi= ip=unknown-ip-addr cmd=get_table : db=yt tbl=stocks3 17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for subqueries 17/08/04 14:59:08 INFO parse.CalcitePlanner: Get metadata for destination tables 17/08/04 14:59:09 INFO ql.Context: New scratch dir is hdfs://localhost:9000/tmp/hive//b5e3e924-e46a-481c-8aef-30d48605a2da/hive_2017-08-04_14-59-08_935_8316159022041430928-1 17/08/04 14:59:09 INFO parse.CalcitePlanner: Completed getting MetaData in Semantic Analysis 17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for source tables 17/08/04 14:59:09 INFO metastore.HiveMetaStore: 0: get_table : db=yt tbl=stocks3 17/08/04 14:59:09 INFO HiveMetaStore.audit: ugi= ip=unknown-ip-addr cmd=get_table : db=yt tbl=stocks3 17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for subqueries 17/08/04 14:59:09 INFO parse.CalcitePlanner: Get metadata for
Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Thank you for the response.
If I insert data from hive directly using below query, select query works
fine.
insert into table stocks PARTITION (years=2004,months=12,days=3)
values('AAPL',1501236980,120.34);
I think the issue here is that when we insert data using IGFS api (append
method), select fails to return the results but if we use insert query,
partitions are created and select query works fine. How to resolve this? Is
partition supported through IGFS?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725p15983.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Hi Mikhail, Thanks for your response. I changed hdfs://localhost:9000 to igfs://localhost:9000 in the core-site.xml of hive home directory but core-site.xml of Hadoop home was still pointing to hdfs://localhost:9000. Issue persists. If I create partition using the insert command on hive I am getting rows on select query. Even after stopping all ignite nodes I am still getting the data on select query. Does this mean that hive is still pointing to hdfs? What else can we check here? Thanks! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725p15937.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Ignite-hadoop accelerator is configured as https://apacheignite.readme.io/v1.0/docs/hadoop-accelerator#section-secondary-file-system. Idea is to run hive queries on IGFS but looks like its not working that way. Not sure how to confirm if hive is connecting to HDFS or IGFS. Also, I think creating partition by passing comma separated string to fs.append is not creating partition correctly which is resulting in no result on hive query. Below are some of the configurations I have. Please let me know if I need to share any more details. core-site.xml is having these properties: fs.defaultFS hdfs://localhost:9000 fs.igfs.impl org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem fs.AbstractFileSystem.igfs.impl org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem dfs.client.block.write.replace-datanode-on-failure.policy NEVER ignite's default-config has below configuration D:/hadoop/etc/hadoop/core-site.xml -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725p15887.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
how to append data to IGFS so that data gets saved to Hive partitioned table?
I have a Hive partitioned table as below: create table stocks (stock string, time timestamp, price float) PARTITIONED BY (years bigint, months bigint, days bigint) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; Data is getting saved in HDFS through IGFS when I pass comma separated string to the fs.append method but when I query on Hive there is no row coming. If I remove the partition while creating the table, I am able to fetch rows from hive. What could be wrong here? How can we save data to a partitioned table through IGFS? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
