Re: Assigning resources to individual MR jobs of a Pig script

s://issues.apache.org/jira/browse/PIG-4424 > > This is blocked by https://issues.apache.org/jira/browse/PIG-2597 and > assigned to me. > > I need to find time to get to this. Sorry! > Koji > > > > On Sunday, March 19, 2017, 10:35:55 AM EDT, Mohammad Tariq < > donta

Re: Assigning resources to individual MR jobs of a Pig script

asic.html > > > Best Regard, > Jeff Zhang > > > > > > On 3/18/17, 6:03 PM, "Mohammad Tariq" wrote: > > >Hi Jeff, > > > >Thank you for the prompt response. However, I can't use Tez because of > >some > >reasons.

Re: Assigning resources to individual MR jobs of a Pig script

//pig.apache.org/docs/r0.16.0/perf.html#enable-tez > > > > > Best Regard, > Jeff Zhang > > > > > > On 3/17/17, 3:24 AM, "Mohammad Tariq" wrote: > > >Hi group, > > > >In any real world pig script we end up with multiple MR jobs(well, most of

Assigning resources to individual MR jobs of a Pig script

Hi group, In any real world pig script we end up with multiple MR jobs(well, most of the times). I was wondering if it's possible to allocate resources to individual MR jobs rather than assigning them at the script level itself. Tried looking at multiple places. Would really appreciate some pointe

Re: Error in Pig Exection

Hi Kiran, Does hdfs://localhost:9000/user/hduser/pig/planet.osm exist on your HDFS? [image: http://] Tariq, Mohammad about.me/mti [image: http://] On Wed, Jun 22, 2016 at 1:08 PM, Kirandeep Kaur wrote: > Dear Sir > > I am working on pig with hadoop as a researcher. I

Re: store to defined filename

Hi there, You could do that with the help of MultipleOutputFormat<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/lib/MultipleOutputFormat.html>class. It extends FileOutputFormat,and allows us to write the output data to different output files. *Warm regards,* *Mohammad

Re: Bulk load in hbase using pig

Could you please let us know how exactly you want to parse your logs? Warm Regards, Tariq cloudfront.blogspot.com On Wed, Feb 26, 2014 at 6:25 PM, David McNelis wrote: > The big question is how the log file needs to be parsed / formatting. I'd > be inclined to write a UDF that would take the

Re: Reading Kafka directly from Pig?

Great job. +1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, Aug 7, 2013 at 8:27 PM, Russell Jurney wrote: > Cool stuff, a Pig Kafka UDF. > > Russell Jurney http://datasyndrome.com > > Begin forwarded message: > > *From:* David Arthur > *Date:* August 7, 2013, 7:41:30 AM PDT > *To:* us...

Re: union

flatten(C); store E into '/dir'; Warm Regards, Tariq cloudfront.blogspot.com On Thu, Jul 25, 2013 at 12:52 PM, Mohammad Tariq wrote: > Hello Keren, > > There is nothing wrong in this. One dataset in Hadoop is usually one > folder and not one file. Pig is doing what it is

Re: union

Hello Keren, There is nothing wrong in this. One dataset in Hadoop is usually one folder and not one file. Pig is doing what it is supposed to do and performing a union on both the files. You would have seen the content of both the files together while doing dump C. Since this is a map only job,

Pig giving priority to "non" Apache Hadoop

Hello list, Today I started Pig on my personal machine after a few weeks to give 0.11.1 a try. As soon as I issued bin/pig it threw this message on my terminal : apache@hadoop:/hadoop/projects/pig-0.11.1$ bin/pig 2013-06-26 06:05:45,121 [main] INFO org.apache.pig.Main - Apache Pig versi

Re: BinCond

Hello Soniya, It's like the ternary or the conditional operator available in java and works just like that. Here is the example provided in the reference manual : *Suppose we have relation A.* A = LOAD 'data' AS (f1:int, f2:int, B:bag{T:tuple(t1:int,t2:int)}); DUMP A; (10,1,{(2,3),(4,6)}

Re: Coding standards of Pig

Sorry for the typo. I missed the second link. http://blog.cloudera.com/blog/2012/08/process-a-million-songs-with-apache-pig/ Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Sun, Apr 21, 2013 at 7:35 PM, Mohammad Tariq wrote: > Hello Raj, > > You might find th

Re: Coding standards of Pig

Hello Raj, You might find these links useful : http://wiki.apache.org/pig/PigPerformance http://wiki.apache.org/pig/PigPerformance And I don't have much idea as far as coding conventions are concerned as I haven't seen much on that. I had come across this small section on the reference manual t

Re: Is there a way to limit the number of maps produced by HBaseStorage ?

Hello Vincent, The number of map tasks for a job is primarily governed by the InputSplits and the InputFormat you are using. So setting it through a config parameter doesn't guarantee that your job would have the specified number of map tasks. However, you can give it a try by using "set

Re: Limit number of Streaming Programs

Folks on the list need some time mate. I have specified a couple of links on the other thread of yours. Check it out and see if it helps. Best Regards, Tariq +91-9741563634 https://mtariq.jux.com/ On Tue, Dec 25, 2012 at 11:09 AM, Kshiva Kps wrote: > Hi, > > Is there any PIG editors and where

Re: Sequence File processing

+1 Best Regards, Tariq +91-9741563634 https://mtariq.jux.com/ On Tue, Dec 25, 2012 at 3:07 AM, Cheolsoo Park wrote: > Hi Srini, > > You can use STRSPLIT to split your "value" chararray and define schema in a > FOREACH. For example, if the "value" consists of 3 integers (i.e. "1|2|3"), > > A= LO

Re: Is Programming Pig book outdated?

Agree with Mr. Jagat. Regards, Mohammad Tariq On Fri, Nov 16, 2012 at 3:26 PM, Jagat Singh wrote: > In open source community no book can be ever latest , so we have to live by > this :) > > I would suggest you to start from this book and see the latest > documentation on p

Re: accessing like array

load the data into a relation and use 'generate' to take only the required fields from this relation and put into some other relation..then store the 2nd relation into some file. Regards, Mohammad Tariq On Tue, Nov 6, 2012 at 7:43 PM, jamal sasha wrote: > Hi, > I h

Re: Unable to store data into HBase

), it is working perfectly fine. It seems there is some compatibility issue between Pig(0.10.0) and HBase(0.92.1). Any comments or suggestions? Regards, Mohammad Tariq On Mon, Sep 3, 2012 at 8:07 PM, chethan wrote: > STORE raw_data INTO ‘hbase://sample_name

Re: Unable to store data into HBase

I don't think there is any problem with that as I am able to execute other queries, like loading data from an HBase table and storing it into another HBase table. Regards, Mohammad Tariq On Mon, Sep 3, 2012 at 1:57 PM, shashwat shriparv wrote: > What can conclude from the error

Unable to store data into HBase

d to read data from "/mapin/testdata2.csv/part-m-0, when I already have data in relation 'z'. Any help would be much appreciated. Many thanks. Regards, Mohammad Tariq

Re: FileAlreadyExistsException while running pig

Hello Haitao, Each time we run a MapReduce job, the job expects the output to be non-existent. If the output path is already there then FileAlreadyExists exception is thrown. And as we know that each Pig job is eventually a MapReduce job, it also expects the same. Regards, Mohammad

Re: foreach in PIG is not working.

re grunt> dump a; grunt> b = foreach a generate name; grunt> dump b; Regards, Mohammad Tariq On Thu, Jul 26, 2012 at 12:06 AM, Mohammad Tariq wrote: > Don't use (',')...By default, LOAD looks for our data in a > tab-delimited file using the default LOAD functio

Re: foreach in PIG is not working.

command properly which I have shown in the previous mail. A=load '/HADOOP/Yogesh/demo.txt' using PigStorage (' ') as (name:chararray, roll:int) Pay attention to "PigStorage (' ')"..I haven't used comma there. Use it..It should work. Regards, Mo

Re: foreach in PIG is not working.

try this : A=load '/HADOOP/Yogesh/demo.txt' using PigStorage (' ') as (name:chararray, roll:int) Regards, Mohammad Tariq On Wed, Jul 25, 2012 at 11:47 PM, pablomar wrote: > are the commas in your file in the write places ? > your DUMP A shows > (Yogesh 1

Re: foreach in PIG is not working.

ry to load the file properly, and you are good to go. Regards, Mohammad Tariq On Wed, Jul 25, 2012 at 11:36 PM, wrote: > Hi all, > > I loaded a file to pig by command from HDFS. > > A=load '/HADOOP/Yogesh/demo.txt' > as (name:chararray, roll:int); > > its get

Re: DATA not storing as comma-separted

Hello Yogesh, Also add these lines, export PIG_CLASSPATH=/HADOOP_HOME/conf & export HADOOP_CONF_DIR=/HADOOP_HOME/conf, and see if it works for you. Regards, Mohammad Tariq On Wed, Jul 25, 2012 at 6:01 PM, wrote: > Hi mohammad, > > when I try the command > > Pig

Re: DATA not storing as comma-separted

Also, it would be help to go to the MapReduce web UI and having a look at the details of the job corresponding to this query. Regards, Mohammad Tariq On Wed, Jul 25, 2012 at 5:31 PM, Mohammad Tariq wrote: > I have worked with pig-0.7.0 once and it was working fine. Try to see > if th

Re: DATA not storing as comma-separted

I have worked with pig-0.7.0 once and it was working fine. Try to see if there is anything interesting in the log files. Also, if possible, share 2-3 lines of your file..I'll give it a try on my machine. Regards, Mohammad Tariq On Wed, Jul 25, 2012 at 5:20 PM, wrote: > Hi Mohamma

Re: DATA not storing as comma-separted

Hi Yogesh, Is 'load' working fine with PigStorage()?? Try to load something using PigStorage(',') and dump it to see if that is working. Regards, Mohammad Tariq On Wed, Jul 25, 2012 at 4:41 PM, wrote: > Hello Dmitriy, > > I have also perfor

Re: Is there a loader that loads a file as a line?

Hello Jonathan, Have a look at Hadoop's WholeFileInputFormat..Might fit into your requirements. Regards,     Mohammad Tariq On Fri, Jun 22, 2012 at 3:39 AM, Prashant Kommireddi wrote: > I think you will need to implement a RecordReader/InputFormat of your own > for this and

Re: Re: How pig get hadoop and hbase configuration?

Could you send me your hadoop and hbase config files??? Regards,     Mohammad Tariq On Wed, Jun 13, 2012 at 1:18 PM, Mohammad Tariq wrote: > "HBase is able to connect to ZooKeeper but the connection closes > immediately." - This error means that your HMaster is not able t

Re: Re: How pig get hadoop and hbase configuration?

"HBase is able to connect to ZooKeeper but the connection closes immediately." - This error means that your HMaster is not able to talk to your Namenode. Regards,     Mohammad Tariq On Wed, Jun 13, 2012 at 1:12 PM, lulynn_2008 wrote: > Hello, > hadoop-core-*.jar and commons-c

Re: How pig get hadoop and hbase configuration?

Hello, Copy the hadoop-core-*.jar from your hadoop folder to the hbase/lib folder.Also copy commons-configuration-1.6.jar from hadoop/lib folder to hbase/lib folder...Some times due to incompatible jars this may happen..do it and see if it works for you. Regards,     Mohammad Tariq On Wed

Re: How to use TOP?

Yes, it would be better if I do it at the time of insertion.Just have to add one more column.Thanks again. Regards,     Mohammad Tariq On Tue, May 22, 2012 at 2:36 PM, Abhinav Neelam wrote: > Doing it in the pig script is not feasible because pig doesn't have any > notion of sequent

Re: How to use TOP?

cript??Many thanks. Regards,     Mohammad Tariq On Tue, May 22, 2012 at 1:16 AM, Abhinav Neelam wrote: > Hey Mohammad, > > You need to have sorting requirements when you say 'top 5' records. Because > relations/bags in Pig are unordered, it's natural to ask: 'top 5

Re: How to use TOP?

Hi Ruslan, Thanks for the response.I think I have made a mistake.Actually I just want the top 5 records each time.I don't have any sorting requirements. Regards,     Mohammad Tariq On Mon, May 21, 2012 at 9:31 PM, Ruslan Al-fakikh wrote: > Hey Mohammad, > > Here > c =

How to use TOP?

b = foreach a { c = TOP(5,3,a); >> generate flatten(c); >> } Could anyone tell me how to achieve thatMany thanks. Regards,     Mohammad Tariq

Re: how to achieve foreach 'n'

better way to do this.I am new to pig so finding it a bit tricky.Many thanks. Regards,     Mohammad Tariq On Thu, May 17, 2012 at 1:44 PM, Mohammad Tariq wrote: > Hello list, > >    I have loaded data from an Hbase table and the relation looks like this - > > 18.98   2000    1.21

how to achieve foreach 'n'

��� �\55.55 2001.5 33.03 656.56 2.8260.76 � ��ȫ ��� �P156.14 200235.52 83.75 2.6 59.57 � ��ȫ ��� �D138.77 2002.5 21.51 105.76 2.6285.89 � ��ȫ ��� �871.89 200327.79 709.01 2.6385.44 Regards,     Mohammad Tariq