Re: [R] Importing Big data to R

[Peter Dalgaard]

> On 12 Jul 2017, at 08:44 , Mangalani Peter Makananisa 
>  wrote:
> 
> Dear R-Gurus,
> 
> I am trying to read in data with 24,349,113 rows to R-3.3.3 (64 bit) and have 
> used  the library   "data.table"  and It managed to read 23,347,070 rows  and 
> the remainder was 2,043 rows only.

I make that 1002043 rows... (typo?). One possibility is that you are skipping 
lines because of something like quote characters in data or some input lines 
being one field short. You might also simply be running out of memory.

However,

1) data.table is a contributed package. Its maintainer may be a better point of 
contact.

2) if you hope to get advice from the R community at large, it would be useful 
to tell us what you did, what kind of data you try to read using which command, 
etc. And, as Jim Holtman already said: error messages, if any. 

-pd

> 
> Could you please advise me as to which library/R-commands is suitable to read 
> the full data in to R?
> 
> Kind regards,
> 
> Mangalani Peter Makananisa (5786)
> South African Revenue Service (SARS)
> +2782 456 4669 / +2712 422 7357
> 
> Please Note: This email and its contents are subject to our email legal 
> notice which can be viewed at 
> http://www.sars.gov.za/Pages/Email-disclaimer.aspx
> 
>   [[alternative HTML version deleted]]
> 
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

-- 
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Office: A 4.23
Email: pd@cbs.dk  Priv: pda...@gmail.com

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

Re: [R] Importing Big data to R

[jim holtman]

A little more information would be useful.  Why did it stop? Was there an
error message?  Can you show the commands/console log of what you did.
Provide information on how much memory your computer has on it.  When the
operation completed, how much memory was used.  An important aspect is how
many columns did the data have.  How big was the file on disk.  What other
objects were in memory at the same time.  The list can go on and on, so
more information would be useful to understand the problem.


Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.

On Wed, Jul 12, 2017 at 2:44 AM, Mangalani Peter Makananisa <
pmakanan...@sars.gov.za> wrote:

> Dear R-Gurus,
>
> I am trying to read in data with 24,349,113 rows to R-3.3.3 (64 bit) and
> have used  the library   "data.table"  and It managed to read 23,347,070
> rows  and the remainder was 2,043 rows only.
>
> Could you please advise me as to which library/R-commands is suitable to
> read the full data in to R?
>
> Kind regards,
>
> Mangalani Peter Makananisa (5786)
> South African Revenue Service (SARS)
> +2782 456 4669 / +2712 422 7357
>
> Please Note: This email and its contents are subject to our email legal
> notice which can be viewed at http://www.sars.gov.za/Pages/
> Email-disclaimer.aspx
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/
> posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[R] Importing Big data to R

[Mangalani Peter Makananisa]

Dear R-Gurus,

I am trying to read in data with 24,349,113 rows to R-3.3.3 (64 bit) and have 
used  the library   "data.table"  and It managed to read 23,347,070 rows  and 
the remainder was 2,043 rows only.

Could you please advise me as to which library/R-commands is suitable to read 
the full data in to R?

Kind regards,

Mangalani Peter Makananisa (5786)
South African Revenue Service (SARS)
+2782 456 4669 / +2712 422 7357

Please Note: This email and its contents are subject to our email legal notice 
which can be viewed at http://www.sars.gov.za/Pages/Email-disclaimer.aspx

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.