[
https://issues.apache.org/jira/browse/PIG-4512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514479#comment-14514479
]
Rohini Palaniswamy commented on PIG-4512:
-----------------------------------------
bq. I've been doing some tests with Pig & Hive, trying to gain some performance
using the OrcStorage class and his "Predicate Push Down" loader. I can't see
any difference in performance between using OrcStorage and HCatLoader.
How is the Pig (OrcStorage or HCatLoader) performance compare to Hive?
> No performance improvement using OrcStorage
> -------------------------------------------
>
> Key: PIG-4512
> URL: https://issues.apache.org/jira/browse/PIG-4512
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.14.0
> Environment: Hortonworks 2.2, Pig 14.0, Hive 0.14.0, Tez
> Reporter: Ángel Álvarez
> Priority: Minor
>
> I've been doing some tests with Pig & Hive, trying to gain some performance
> using the OrcStorage class and his "Predicate Push Down" loader. I've
> followed the next steps:
> 1, Download a dataset
> ftp://ita.ee.lbl.gov/traces/NASA_access_log_Aug95.gz
> 2. Create a new larger file by copying the same original file multiple times.
> cat NASA_access_log_Aug95 NASA_access_log_Aug95 ... > NASA
> 3. Add a new line in the data file.
> echo 'slppp6.intermind.net - - [01/Aug/1995:00:00:11 -0400] "GET test
> HTTP/1.0" 200 9202' >> NASA
> 4. Create the ORC table in Hive
> DROP TABLE nasadata_txt;
> DROP TABLE nasadata_orc;
> CREATE TABLE nasadata_txt(ip VARCHAR(50), user_identifier VARCHAR(50),
> user_id VARCHAR(50),date_time VARCHAR(50),zone VARCHAR(10),method
> VARCHAR(5),uri VARCHAR(200),version VARCHAR(10),status DECIMAL(3,0),size
> DECIMAL(10,0)) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' STORED AS
> TEXTFILE;
> CREATE TABLE nasadata_orc(ip VARCHAR(50), user_identifier VARCHAR(50),
> user_id VARCHAR(50),date_time VARCHAR(50),zone VARCHAR(10),method
> VARCHAR(5),uri VARCHAR(200),version VARCHAR(10),status DECIMAL(3,0),size
> DECIMAL(10,0)) STORED AS ORC;
> -- Load into Text table
> LOAD DATA LOCAL INPATH 'NASA_access_log_Aug95' INTO TABLE nasadata_txt;
> LOAD DATA LOCAL INPATH 'NASA.*' INTO TABLE nasadata_txt;
> -- Copy to ORC table
> INSERT OVERWRITE TABLE nasadata_orc SELECT * FROM nasadata_txt;
> 5. Execute this pig script
> rmf /tmp/pruebaPPD;
> A = LOAD '/apps/hive/warehouse/nasadata_orc' using OrcStorage() as
> (ip,user_identifier,user_id,date_time,zone,method,uri,version,status,size);
> A = foreach A generate ip,uri,status;
> A = filter A by uri == 'test';
> A = group A by uri;
> A = foreach A generate group,COUNT(*);
> store A into '/tmp/pruebaPPD' using PigStorage(';');
> 6. Execute the previous script replacing OrcStorage by
> org.apache.hive.hcatalog.pig.HCatLoader.
> I can't see any difference in performance between using OrcStorage and
> HCatLoader. Is there anything wrong in what I'm doing? Do I have to set any
> property?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
