Reading a files in a folder

[Geetha Weerasooriya]

Hi ,
 
I have data files which are in different folders. I want to run the same
program on all the data files. For example I have 12 folders for 12
months of the year and each folder contains the files for each day of
the month. So one file contains 30 or 31 files. 
 
Can some one kindly guide me to do this? Or can you give any reference
material to study this type of data handling?
 
Thanks in advance.
 
Best wishes,
 
Geetha

incorrect output

[Geetha Weerasooriya]

Hi,
 
I am extremely grateful to all those who answered my question and I am
happy to say that I was able to get a solution to my problem from the
suggestions and explanations given by you. I understood my mistakes and
learned a lot from you. 
 
Thanks again.
 
Best wishes,
 
Geetha
 

incorrect output

[Geetha Weerasooriya]

Hi dear all,
 
I have written a script which reads the data lines and when a blank line
is found all the data up to the previous line is considered as a one set
and new set is started after the next line. Then I subtracted from one
element of the last line of that set the same element of the first line
of data.  When I run this program, the trajectory travel time calculated
is correct only for the first 3 trajectories and there after it is
wrong. I can't understand why it gives correct value for first three
trajectories. Can someone please help me??
My data file looks like follows:

5/1/2004
 07:06:43
10
139.6668
35.52782
21.2
32952056
5593
0.86
25603
3
The script is as follows:
 
#!perl/bin/perl ;
use strict;
#use warnings;
my $i;
my $j;
my $data;
my @line;
my @time_in_seconds;
my $travel_time;
my $trajectory_travel_time;
my @vid;
open (FILE, " data.csv" ) || die " can't open the file";
open (OUT1, "> travel_time.csv") || die " can't open the file";
open (OUT2, "> trajectory.csv") || die " can't open the file";
$i = 0;
$j=1;
while () {
$i++;
   $data = $_;
   chomp;
   @line=split /,/;
   $vid[$i] = $line[2];
 
if (@line != ()) {
   $time_in_seconds[$i] = $line[-2];
  if  ( $i==1) {
 $travel_time= 0;
  } else {

  $travel_time = $time_in_seconds[$i] - $time_in_seconds[1];
  } 
 
  print OUT1 "$_,$travel_time \n";
 
} else {
$trajectory_travel_time = $time_in_seconds[-1] -
$time_in_seconds[1];
   print OUT2 "$vid[$i-1],Trajectory$j, $trajectory_travel_time
\n";
   $j++;
   $i=0;
   print OUT1 "\n";
}
}
 
When I turn on the " use warnings " it gives the warning ' Use  of
uninitialized value in numeric ne(!=) at . line 37,  line..
 
Here line 37 is" if (@line != ()) {"
 
Kind regards, 
Geetha
   
 

Access elements of a array of hases

[Geetha Weerasooriya]

Hi ,
 
I have an Array of Hashes   as follows:
 
@array = (
 { 'A'=>1, 'B' =>2, 'C'=>3, 'D'=>4}
{ 'A'=>5, 'B' =>6, 'C'=>7, 'D'=>8}
{ 'A'=>9, 'B' =>10, 'C'=>11, 'D'=>12}
{ 'A'=>13, 'B' =>14, 'C'=>15, 'D'=>16}
{ 'A'=>17, 'B' =>18, 'C'=>19, 'D'=>20}
{ 'A'=>21, 'B' =>22, 'C'=>23, 'D'=>24}
)
 
I have given a reference to this array in my script.
 
I want to do some calculation with only one key of each hash. For
example, I want to subtract from the value of key 'B' of last hash all
the other values of the same key into a one column like follows 
 
22-2  20   
22-6  16
22-10 12
22-14  8
22-18  4
22-22  0
 
Can you please tell me how to do this ? 
 
Kind regards,
 
Geetha
 

Remving unnecessary data

[Geetha Weerasooriya]

Dear Mr. Dani,
 
Thank you so much for the perl script and   for your kindness. It seems
you have taken great trouble to help me. It will be very much helpful to
me. I am really thankful to you.
 
I will try that script and let you know the result.
 
Thanks again,
 
Kind regards,
 
Geetha

Removing Unnecessary data

[Geetha Weerasooriya]

Dear Mr. Dani,

 

Thanks for your mail. I am so happy that you are exactly understanding
my problem and you have already understood it correctly even beyond what I have
explained.  

 

Sorry there are 10 data fields and I have given the 10 labels.
Distance and Flag are two labels. Distance means distance from the origin of the
route(Point A) to the bus location.

 

When calculating the sequence, I have considered only the
Distance data. As explained in my previous mail, depending on the distance I
have given numbers 0,1,2,3 and 4 and which is in last column.

 In this case I
am considering only one route and hence route no need not be considered. As you
have correctly understood, I have to consider the Date and VID also in
recognizing the sequence.

 

Yes, it is true that the bus has moved along the route
several times in a day. Sometimes more than 20 times.  Still I am thinking how to separate these
drives. I want to give them a name such as Trajectory1, Trajectory 2 , so on.

 

The sequence is exactly  0,0,0,0,……0, 1,1,1,  1, 2,2,..2,
3,3,.3, 4,4,…..4.
like that. 
As I have observed one sequence has in average 1500 records, because bus
stops at bus halts and intersections and then same recoding is there for a long
time. Since the number of records per group depends on the speed of the bus,
bus stops ect, there is no
maximum number of records per group.

 

Yes the group of records from which I calculate the sequence
is ordered in the file.

 

Yes I have the 3 cases you have identified. But the case is
when the bus goes to the end of the route it change the direction and then we
get 0,1,2,3,4,3,2,1,0,1,2,…that way. 

 

I will send here 30 lines of my data set, but unfortunately
it is difficult to cover a sequence in 30 lines.( it
takes at least 1200 lines)

 

Thank you very much for taking trouble in this connection.

 

Best wishes,

 

Geetha








data_sample.xls
Description: MS-Excel spreadsheet
-- 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
 

RE: Removing unnecessary data

[Geetha Weerasooriya]

Dear Mr.Dani,

Thank you very much for the reply. I understand that but question is not
clear. I will explain my problem little more.

Below is the sample of my data set. Actual data file is very much
larger.( about 3 GB)


DateTime  Veh IDLongitude   Latitude Speed Odometer
Route No
distance flag
 2003/11/12 8:32:43 10  139.6368501 35.51527949 23.6
27406416 1  21773
 2003/11/12 8:32:44 10  139.6368501 35.51527949 23.6
27406416 1  21773
 2003/11/12 8:32:45 10  139.636606  35.51526727 27.6
27406436 1  21553

The Points A,B and C are defined.  In other words, the coordinates of
these points are known.

I tried to attach a flag depending on the distance to the bus  from the
origin of the route as follows:

if the distance from origin  is 0 to 50, 0
if it is 50 to 2000, 1 
if it is 2000 to 2070, 2 
if it is 2070 to 5775 , 3
if  5775 to 5830 flag is 4.

Point A(origin) lies between 0-50
Point B(destination) lies between 5775-5830
Point C(turning to other route) lies between 2000-2070 

If bus has gone from origin to destination we get a series of 0,1,2,3
and 4. 
If bus is coming other direction, sequence is 4,3,2,1 and 0. I want to
separate these two data into two files. If it has turned at point C we
get sequence of 0,1 and 2 only. This data I don't need.

My problem is how to split this based on the flag I have already
attached.

Please apologize me for the long mail.

Kind regards,

Geetha

-Original Message-
From: D. Bolliger [mailto:[EMAIL PROTECTED] 
Sent: Sunday, September 03, 2006 7:11 PM
To: beginners@perl.org
Subject: Re: Removing unnecessary data

Geetha Weerasooriya am Sonntag, 3. September 2006 08:19:
> Hi Dear all,

Hi Geetha Weerasooriya

> I have a data file for bus trajectories where bus location at each
> second along the route is given. The specified route for the bus is
from
> origin (Point A) to Destination (Point B) , but sometimes bus has
> changed the route at  a point (Point C) in between A and B and has
gone
> in some other route.   I want to remove these data lines relevant to A
> to C when the bus changes the route. In other words, I want to keep
only
> the data where bus had completed the trajectory from Point A to B.
>
> Can some one please help me? I am still new to perl.

It shurely can be done with perl, but I think you have to provide more
info so 
somebody can help:

How does the data look and what's its meaning? (How can the points A/B/C
be 
identified?)
What did you try so far?

Dani

-- 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
<http://learn.perl.org/> <http://learn.perl.org/first-response>



-- 
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
<http://learn.perl.org/> <http://learn.perl.org/first-response>

Removing unnecessary data

[Geetha Weerasooriya]

Hi Dear all,
 
I have a data file for bus trajectories where bus location at each
second along the route is given. The specified route for the bus is from
origin (Point A) to Destination (Point B) , but sometimes bus has
changed the route at  a point (Point C) in between A and B and has gone
in some other route.   I want to remove these data lines relevant to A
to C when the bus changes the route. In other words, I want to keep only
the data where bus had completed the trajectory from Point A to B.
 
Can some one please help me? I am still new to perl.
 
 Thanks in advance,
 
Kind regards,
Geetha
 
 

Seperate into different files

[Geetha Weerasooriya]

Hi dear all,
 
I have a data file where there are bus probe data for different bus
ID's. I want to separate these data based on vehicle ID, date,  time and
distance from intersections. Can some one kindly give me an idea to do
this?
 
Kind regards,
 
Geetha

error message

[Geetha Weerasooriya]

Dear All,
 
In my perl code for MapMatching, I have following while loop. When I run
the program I get the following two error messages(for each data line in
the data file)
But the out put file is created.
 
Use of uninitialized value in int at filename.pl lineNo,  line ..
Use of unititialized value in multiplication(*) at filename.pl lineNo,
 line ..
 
The errors refer to the lines  in bold in the code
 
Can you please  teach me where the error is?
 
Kind regards,
 
Geetha
 
 
 
while()  {
chop;
s/\s//g;
 
/\d+\/\d+\/\d+,\d+:\d+:\d+,\d+,(\d+\.*\d*),(\d+\.*\d*),\d+,\d+,\d+,\d+\.
*\d*,\d+$/
or die "Error in mtchSq2Route.pl! bad format in
SQ record.\n$_\n";
$lngi_bgn = $1;
$lati_bgn = $2;
   
 my ($rt1,$dist1,$pos1) = &getFootPointDistToRoot($lngi_bgn,
$lati_bgn);

 $pos1 = int($pos1);
 $dist1 = int($dist1*100)/100;
  
print "$_,$rt->{ID},$pos1,$dist1\n";
 

error message

[Geetha Weerasooriya]

Dear All,
 
In my perl code for MapMatching, I have following while loop. When I run
the program I get the following two error messages(for each data line in
the data file)
But the out put file is created.
 
Use of uninitialized value in int at filename.pl lineNo,  line ..
Use of unititialized value in multiplication(*) at filename.pl lineNo,
 line ..
 
The errors refer to the lines  in bold in the code
 
Can you please  teach me where the error is?
 
Kind regards,
 
Geetha
 
 
 
while()  {
chop;
s/\s//g;
 
/\d+\/\d+\/\d+,\d+:\d+:\d+,\d+,(\d+\.*\d*),(\d+\.*\d*),\d+,\d+,\d+,\d+\.
*\d*,\d+$/
or die "Error in mtchSq2Route.pl! bad format in
SQ record.\n$_\n";
$lngi_bgn = $1;
$lati_bgn = $2;
   
 my ($rt1,$dist1,$pos1) = &getFootPointDistToRoot($lngi_bgn,
$lati_bgn);

 $pos1 = int($pos1);
 $dist1 = int($dist1*100)/100;
  
print "$_,$rt1->{ID},$pos1,$dist1\n";
 

Next

[Geetha Weerasooriya]

Dear All,
 
Thank you so much for sending me the solution to my problem. It was the
first problem I asked and I am really happy I got very well explained
answers. I understood it well.
 
Thanks again for every one who take time to answer my question.
 
Kind regards,
 
Geetha

Next

[Geetha Weerasooriya]

Dear all,
 
When I was reading a Perl code I found the following line. Can u please
explain what it means?
 
!defined($rt_nearest) or $dh<$dist or next;
 
Kind regards,
 
Geetha