RE: BUG #15858: could not stat file - over 4GB

2021-02-24 Thread wangsh.f...@fujitsu.com
Thank you for sharing

Best regards
Shenhao Wang

-Original Message-
From: Michael Paquier  
Sent: Thursday, February 25, 2021 2:22 PM
To: Wang, Shenhao/王 申豪 
Cc: Tom Lane ; Juan José Santamaría Flecha 
; Emil Iggland ; PostgreSQL 
Hackers 
Subject: Re: BUG #15858: could not stat file - over 4GB

On Thu, Feb 25, 2021 at 06:07:06AM +, wangsh.f...@fujitsu.com wrote:
> I noticed that this modification only commit into master branch, 
> there is still have a problem on 12.6 or 13.2 on Windows.
> 
> Do you have a plan to backpatch this commit into REL_12_STABLE or 
> REL_13_STABLE ?

The change to be able to fix that stuff is invasive.  So, while I
don't really object to a backpatch of this change in the future, I
think that it would be wiser to wait until we get more feedback with
the release of Postgres 14 before doing a backpatch to older
versions.  So we are in a wait phase for the moment.

Thanks,
--
Michael


Re: BUG #15858: could not stat file - over 4GB

2021-02-24 Thread Michael Paquier
On Thu, Feb 25, 2021 at 06:07:06AM +, wangsh.f...@fujitsu.com wrote:
> I noticed that this modification only commit into master branch, 
> there is still have a problem on 12.6 or 13.2 on Windows.
> 
> Do you have a plan to backpatch this commit into REL_12_STABLE or 
> REL_13_STABLE ?

The change to be able to fix that stuff is invasive.  So, while I
don't really object to a backpatch of this change in the future, I
think that it would be wiser to wait until we get more feedback with
the release of Postgres 14 before doing a backpatch to older
versions.  So we are in a wait phase for the moment.

Thanks,
--
Michael


signature.asc
Description: PGP signature


Re: BUG #15858: could not stat file - over 4GB

2021-02-24 Thread Tom Lane
"wangsh.f...@fujitsu.com"  writes:
> Do you have a plan to backpatch this commit into REL_12_STABLE or 
> REL_13_STABLE ?

https://www.postgresql.org/message-id/ycszix2a2ilsv...@paquier.xyz

regards, tom lane




RE: BUG #15858: could not stat file - over 4GB

2021-02-24 Thread wangsh.f...@fujitsu.com
Hi,

I noticed that this modification only commit into master branch, 
there is still have a problem on 12.6 or 13.2 on Windows.

Do you have a plan to backpatch this commit into REL_12_STABLE or REL_13_STABLE 
?

The commit:
https://github.com/postgres/postgres/commit/bed90759fcbcd72d4d06969eebab81e47326f9a2
https://github.com/postgres/postgres/commit/ed30b1a60dadf2b7cc58bce5009ad8676b8fe479


--
Best regards
Shenhao Wang


Re: BUG #15858: could not stat file - over 4GB

2020-10-12 Thread Michael Paquier
On Mon, Oct 12, 2020 at 11:13:38AM -0400, Tom Lane wrote:
> Juan José Santamaría Flecha wrote:
>> Uhm, a good question indeed, forcing errno serves no purpose there.
> 
> OK, changed.

Thanks!
--
Michael


signature.asc
Description: PGP signature


Re: BUG #15858: could not stat file - over 4GB

2020-10-12 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?=  
writes:
> On Mon, Oct 12, 2020 at 5:27 AM Tom Lane  wrote:
>> Michael Paquier  writes:
>>> Why are we forcing errno=ENOENT here?  Wouldn't it be correct to use
>>> _dosmaperr(GetLastError()) to get the correct errno?

>> Fair question.  Juan, was there some good reason not to look at
>> GetLastError() in this step?

> Uhm, a good question indeed, forcing errno serves no purpose there.

OK, changed.

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-10-12 Thread Juan José Santamaría Flecha
On Mon, Oct 12, 2020 at 5:27 AM Tom Lane  wrote:

> Michael Paquier  writes:
> > Why are we forcing errno=ENOENT here?  Wouldn't it be correct to use
> > _dosmaperr(GetLastError()) to get the correct errno?
>
> Fair question.  Juan, was there some good reason not to look at
> GetLastError() in this step?
>

Uhm, a good question indeed, forcing errno serves no purpose there.

Regards,

Juan José Santamaría Flecha


Re: BUG #15858: could not stat file - over 4GB

2020-10-11 Thread Tom Lane
Michael Paquier  writes:
> Why are we forcing errno=ENOENT here?  Wouldn't it be correct to use
> _dosmaperr(GetLastError()) to get the correct errno?

Fair question.  Juan, was there some good reason not to look at
GetLastError() in this step?

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-10-11 Thread Michael Paquier
On Sat, Oct 10, 2020 at 08:34:48PM -0400, Tom Lane wrote:
> Nah, I fixed that hours ago (961e07b8c).  jacana must not have run again
> yet.

Indeed, thanks.  I have missed one sync here.

+   hFile = CreateFile(name,
+  GENERIC_READ,
+  (FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE),
+  ,
+  OPEN_EXISTING,
+  (FILE_FLAG_NO_BUFFERING | FILE_FLAG_BACKUP_SEMANTICS |
+   FILE_FLAG_OVERLAPPED),
+  NULL);
+   if (hFile == INVALID_HANDLE_VALUE)
+   {
+   CloseHandle(hFile);
+   errno = ENOENT;
+   return -1;
+   }
Why are we forcing errno=ENOENT here?  Wouldn't it be correct to use
_dosmaperr(GetLastError()) to get the correct errno?  This code would
for example consider as non-existing a file even if we fail getting it
because of ERROR_SHARING_VIOLATION, which should map to EACCES.  This
case can happen with virus scanners taking a non-share handle on files
being looked at in parallel of this code path.
--
Michael


signature.asc
Description: PGP signature


Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Tom Lane
Michael Paquier  writes:
> We are visibly not completely out of the woods yet, jacana is
> reporting a compilation error:

Nah, I fixed that hours ago (961e07b8c).  jacana must not have run again
yet.

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Michael Paquier
On Sat, Oct 10, 2020 at 09:00:27PM +0200, Juan José Santamaría Flecha wrote:
> Great, thanks again to everyone who has taken some time to look into this.

We are visibly not completely out of the woods yet, jacana is
reporting a compilation error:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=jacana=2020-10-10%2018%3A00%3A28
Oct 10 14:04:40
c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/port/win32stat.c:
In function 'fileinfo_to_stat':
Oct 10 14:04:40
c:/mingw/msys/1.0/home/pgrunner/bf/root/HEAD/pgsql.build/../pgsql/src/port/win32stat.c:151:13:
error: 'BY_HANDLE_FILE_INFORMATION {aka struct
_BY_HANDLE_FILE_INFORMATION}' has no member named 'nFileSizeLowi'; did
you mean 'nFileSizeLow'?
Oct 10 14:04:40   fiData.nFileSizeLowi);
Oct 10 14:04:40  ^
Oct 10 14:04:40  nFileSizeLow

I don't have the time to check MinGW and HEAD now, so that's just a
heads-up.
--
Michael


signature.asc
Description: PGP signature


Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Juan José Santamaría Flecha
On Sat, Oct 10, 2020 at 7:43 PM Tom Lane  wrote:

>
> I concur with Michael that it's inappropriate to make an end run around
> _dosmaperr() here.  If you think that the DEBUG5 logging inside that
> is inappropriate, you should propose removing it outright.
>
> Pushed the rest of this.
>

Great, thanks again to everyone who has taken some time to look into this.

Regards,

Juan José Santamaría Flecha


Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?=  
writes:
> If the file does not exist there is no need to call _dosmaperr() and log
> the error.

I concur with Michael that it's inappropriate to make an end run around
_dosmaperr() here.  If you think that the DEBUG5 logging inside that
is inappropriate, you should propose removing it outright.

Pushed the rest of this.

(pgindent behaved differently around PFN_NTQUERYINFORMATIONFILE today
than it did yesterday.  No idea why.)

> The meaningful error should come from the previous call, and an error from
> CloseHandle() could mask it. Not sure it makes a difference anyhow.

Would CloseHandle() really touch errno at all?  But this way is
certainly safer, so done.

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Juan José Santamaría Flecha
On Sat, Oct 10, 2020 at 2:24 PM Michael Paquier  wrote:

>
> -   _dosmaperr(GetLastError());
> +   DWORD   err = GetLastError();
> +
> +   /* report when not ERROR_SUCCESS */
> +   if (err == ERROR_FILE_NOT_FOUND || err ==
> ERROR_PATH_NOT_FOUND)
> +   errno = ENOENT;
> +   else
> +   _dosmaperr(err);
> Why are you changing that?  The original coding is fine, as
> _dosmaperr() already maps ERROR_FILE_NOT_FOUND and
> ERROR_PATH_NOT_FOUND to ENOENT.
>

If the file does not exist there is no need to call _dosmaperr() and log
the error.

>
> -  _dosmaperr(GetLastError());
> +  DWORD   err = GetLastError();
> +
>CloseHandle(hFile);
> +  _dosmaperr(err);
> These parts are indeed incorrect.  CloseHandle() could overwrite
> errno.
>

The meaningful error should come from the previous call, and an error from
CloseHandle() could mask it. Not sure it makes a difference anyhow.

Regards,

Juan José Santamaría Flecha


Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Michael Paquier
On Sat, Oct 10, 2020 at 01:31:21PM +0200, Juan José Santamaría Flecha wrote:
> Thanks for taking care of this. I see no problems in the build farm, but
> please reach me if I missed something.

Thanks for continuing your work on this patch.  I see no related
failures in the buildfarm.

-   _dosmaperr(GetLastError());
+   DWORD   err = GetLastError();
+
+   /* report when not ERROR_SUCCESS */
+   if (err == ERROR_FILE_NOT_FOUND || err == ERROR_PATH_NOT_FOUND)
+   errno = ENOENT;
+   else
+   _dosmaperr(err);
Why are you changing that?  The original coding is fine, as
_dosmaperr() already maps ERROR_FILE_NOT_FOUND and
ERROR_PATH_NOT_FOUND to ENOENT.

-  _dosmaperr(GetLastError());
+  DWORD   err = GetLastError();
+
   CloseHandle(hFile);
+  _dosmaperr(err);
These parts are indeed incorrect.  CloseHandle() could overwrite
errno.
--
Michael


signature.asc
Description: PGP signature


Re: BUG #15858: could not stat file - over 4GB

2020-10-10 Thread Juan José Santamaría Flecha
On Fri, Oct 9, 2020 at 10:22 PM Tom Lane  wrote:

> Emil Iggland  writes:
> > I tested the patch at hand, and it performs as expected. Files larger
> than 4GB can be imported.
>
> Thanks for testing!
>

  Thanks for testing! +1

>
> I'd been expecting one of our Windows-savvy committers to pick this up,
> but since nothing has been happening, I took it on myself to push it.
> I'll probably regret that :-(
>

Thanks for taking care of this. I see no problems in the build farm, but
please reach me if I missed something.

>
> I made a few cosmetic changes, mostly reorganizing comments in a way
> that made more sense to me.
>
> I was working on a new version, which was pgindent-friendlier and clearer
about reporting an error when 'errno' is not informed. Please find attached
a patch with those changes.

Regards,

Juan José Santamaría Flecha


0001-win32stat-indent-and-errormapping-v1.patch
Description: Binary data


Re: BUG #15858: could not stat file - over 4GB

2020-10-09 Thread Tom Lane
Emil Iggland  writes:
> I tested the patch at hand, and it performs as expected. Files larger than 
> 4GB can be imported.

Thanks for testing!

I'd been expecting one of our Windows-savvy committers to pick this up,
but since nothing has been happening, I took it on myself to push it.
I'll probably regret that :-(

I made a few cosmetic changes, mostly reorganizing comments in a way
that made more sense to me.

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-10-07 Thread Emil Iggland
The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:   tested, passed
Spec compliant:   not tested
Documentation:not tested

I tested the patch at hand, and it performs as expected. Files larger than 4GB 
can be imported.

Steps: 
0) create a csv-file that is sufficiently big (>4GB), and one that is small. 
Use these files to test.
1a) Attempt to import the small file using devel-version.
1b) EXPECTED: success, ACTUAL: success
2a) Attempt to import the big file using devel-version.
2b) EXPECTED: failure, ACTUAL: failure
3) Apply patch and build new version
4a) Attempt to import the small file using patched-version.
4b) EXPECTED: success, ACTUAL: success
4a) Attempt to import the big file using patched-version.
4b) EXPECTED: success, ACTUAL: success

The code looks sensible, it is easy to read and follow. The code uses 
appropriate win32 functions to perform the task. 

Code calculates file size using the following method: buf->st_size = ((__int64) 
fiData.nFileSizeHigh) << 32 | (__int64)(fiData.nFileSizeLow);
The hard coded constant 32 is fine, nFileSizeHigh is defined as a DWORD in the 
Win32 API, which is a 32 bit unsigned integer. There is no need to a dynamic 
calculation.

There are minor "nit-picks" that I would change if it were my code, but do not 
change the functionality of the code. 

1) 
if (GetFileAttributes(name) == INVALID_FILE_ATTRIBUTES)
{
  errno = ENOENT;
  return -1;
}

Here I would call _dosmaperr(GetLastError()) instead, just to take account of 
the possibility that some other error occurred.  Following this change there 
are slight inconsistency in the order of "CloseHandle(hFile), errno = ENOENT; 
return -1" and "_dosmaperr(GetLastError()); CloseHandle(hFile); return -1". I 
would prefer consistent ordering, but that is not important.

The new status of this patch is: Ready for Committer


Re: BUG #15858: could not stat file - over 4GB

2020-09-18 Thread Juan José Santamaría Flecha
On Thu, Sep 17, 2020 at 8:47 PM Juan José Santamaría Flecha <
juanjo.santama...@gmail.com> wrote:

> On Thu, Sep 17, 2020 at 6:04 PM Tom Lane  wrote:
>
>>
>> One thing I noticed, which is a pre-existing problem but maybe now
>> is a good time to consider it, is that we're mapping lstat() to be
>> exactly stat() on Windows.  That made sense years ago when (we
>> believed that) Windows didn't have symlinks, but surely it no longer
>> makes sense.
>>
>
> I will have to take a better look at it, but from a quick look it, all
> lstat() calls seem to test just if the file exists, and that can be done
> with a cheap call to GetFileAttributes(). Would a limited (but fast)
> lstat(), where only st_mode could be informed, be acceptable?
>

After thinking more about this, that approach would be problematic for
DELETE_PENDING files. The proposed patch logic is meant to maintain current
behaviour, which is not broken for WIN32 symlinks AFAICT.

Regards,

Juan José Santamaría Flecha


Re: BUG #15858: could not stat file - over 4GB

2020-09-17 Thread Juan José Santamaría Flecha
On Thu, Sep 17, 2020 at 6:04 PM Tom Lane  wrote:

> =?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= <
> juanjo.santama...@gmail.com> writes:
> > Thanks for the reminder. Please find attached a rebased version.
>
> (This hasn't shown up on -hackers yet, maybe caught in moderation?)
>

Thanks for looking into it. Finally, it went through. I will be removing
bug-list from now on.

>
> I took a quick look through this.  I'm not qualified to review the
> actual Windows code in win32stat.c, but as far as the way you're
> plugging it into the system goes, it looks good and seems to comport
> with the discussion so far.
>
> One thing I noticed, which is a pre-existing problem but maybe now
> is a good time to consider it, is that we're mapping lstat() to be
> exactly stat() on Windows.  That made sense years ago when (we
> believed that) Windows didn't have symlinks, but surely it no longer
> makes sense.
>

I will have to take a better look at it, but from a quick look it, all
lstat() calls seem to test just if the file exists, and that can be done
with a cheap call to GetFileAttributes(). Would a limited (but fast)
lstat(), where only st_mode could be informed, be acceptable?

>
> Another more trivial point is that it'd be good to run the new code
> through pgindent before committing.
>

I do not have pgindent in the WIN32 machine, but I will try to for the next
version.

Regards,

Juan José Santamaría Flecha


Re: BUG #15858: could not stat file - over 4GB

2020-09-17 Thread Ranier Vilela
Em qui., 17 de set. de 2020 às 14:37, Juan José Santamaría Flecha <
juanjo.santama...@gmail.com> escreveu:

>
> On Thu, Sep 17, 2020 at 9:46 AM Michael Paquier 
> wrote:
>
>>
>> Could you send a rebase of the patch?  Thanks!
>>
>
> Thanks for the reminder. Please find attached a rebased version.
>
Sorry, I'm missing something?
What's wrong with _stat64?

 Pasta de C:\tmp

18/08/2020  16:51 6.427.512.517 macOS_Catalina.7z
   1 arquivo(s)  6.427.512.517 bytes
   0 pasta(s)   149.691.797.504 bytes disponíveis

C:\usr\src\tests\stat>crt_stat
File size : 6427512517
Drive : C:
Time modified : Tue Aug 18 16:51:47 2020

regards,
Ranier Vilela
// crt_stat.c
// This program uses the _stat function to
// report information about the file named crt_stat.c.

#include 
#include 
#include 
#include 
#include 

int main( void )
{
   struct _stat64 buf;
   int result;
   char timebuf[26];
   char* filename = "c:\\tmp\\macOS_Catalina.7z";
   errno_t err;

   // Get data associated with "crt_stat.c":
   result = _stat64( filename,  );

   // Check if statistics are valid:
   if( result != 0 )
   {
  perror( "Problem getting information" );
  switch (errno)
  {
 case ENOENT:
   printf("File %s not found.\n", filename);
   break;
 case EINVAL:
   printf("Invalid parameter to _stat.\n");
   break;
 default:
   /* Should never be reached. */
   printf("Unexpected error in _stat.\n");
  }
   }
   else
   {
  // Output some of the statistics:
  printf( "File size : %lld\n", buf.st_size );
  printf( "Drive : %c:\n", buf.st_dev + 'A' );
  err = ctime_s(timebuf, 26, _mtime);
  if (err)
  {
 printf("Invalid arguments to ctime_s.");
 exit(1);
  }
  printf( "Time modified : %s", timebuf );
   }
}

Re: BUG #15858: could not stat file - over 4GB

2020-09-17 Thread Tom Lane
Ranier Vilela  writes:
> What's wrong with _stat64?

See upthread.

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-09-17 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?=  
writes:
> Thanks for the reminder. Please find attached a rebased version.

(This hasn't shown up on -hackers yet, maybe caught in moderation?)

I took a quick look through this.  I'm not qualified to review the
actual Windows code in win32stat.c, but as far as the way you're
plugging it into the system goes, it looks good and seems to comport
with the discussion so far.

One thing I noticed, which is a pre-existing problem but maybe now
is a good time to consider it, is that we're mapping lstat() to be
exactly stat() on Windows.  That made sense years ago when (we
believed that) Windows didn't have symlinks, but surely it no longer
makes sense.

Another more trivial point is that it'd be good to run the new code
through pgindent before committing.

regards, tom lane




Re: BUG #15858: could not stat file - over 4GB

2020-09-17 Thread Juan José Santamaría Flecha
On Thu, Sep 17, 2020 at 9:46 AM Michael Paquier  wrote:

>
> Could you send a rebase of the patch?  Thanks!
>

Thanks for the reminder. Please find attached a rebased version.

Regards,

Juan José Santamaría Flecha


0001-Support-for-large-files-on-Win32-v8.patch
Description: Binary data


Re: BUG #15858: could not stat file - over 4GB

2020-09-17 Thread Michael Paquier
On Mon, Oct 28, 2019 at 06:13:58PM +0100, Juan José Santamaría Flecha wrote:
> At this moment is missing review, so it is probably far from being
> commitable. Any attention is appreciated and might help pushing it forward.
> As a personal note, I have to check that is still applies before the
> upcoming commitfest.

Could you send a rebase of the patch?  Thanks!
--
Michael


signature.asc
Description: PGP signature


Re: BUG #15858: could not stat file - over 4GB

2020-09-10 Thread Greg Steiner
I assigned myself as a reviewer for this patch, as I hit this bug today and had 
to perform a workaround.  I have never reviewed a patch before but will try to 
update within the next 5 days.  I intend on performing "Implements Feature" 
reviewing.

Re: BUG #15858: could not stat file - over 4GB

2020-02-29 Thread Juan José Santamaría Flecha
On Sat, Feb 29, 2020 at 9:40 AM Juan José Santamaría Flecha <
juanjo.santama...@gmail.com> wrote:

> On Sat, Feb 29, 2020 at 12:44 AM Tom Lane  wrote:
>
>>
>> The cfbot thinks this doesn't compile on Windows [1].  Looks like perhaps
>> a missing-#include problem?
>
>
> The define logic for _WIN32_WINNT includes testing of _MSC_VER, and is not
> a proper choice for MSVC 2013 as the cfbot is showing.
>

The cfbot is not happy yet. I will backtrack a bit on the cruft cleanup.

Please find attached a new version addressing this issue.

Regards,

Juan José Santamaría Flecha

>
>>
>


0001-Support-for-large-files-on-Win32-v7.patch
Description: Binary data


Re: BUG #15858: could not stat file - over 4GB

2020-02-29 Thread Juan José Santamaría Flecha
On Sat, Feb 29, 2020 at 12:44 AM Tom Lane  wrote:

>
> The cfbot thinks this doesn't compile on Windows [1].  Looks like perhaps
> a missing-#include problem?


The define logic for _WIN32_WINNT includes testing of _MSC_VER, and is not
a proper choice for MSVC 2013 as the cfbot is showing.

Please find attached a new version addressing this issue.

Regards,

Juan José Santamaría Flecha

>
>


0001-Support-for-large-files-on-Win32-v6.patch
Description: Binary data


Re: BUG #15858: could not stat file - over 4GB

2020-02-28 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?=  
writes:
> The latest version of this patch could benefit from an update. Please find
> attached a new version.

The cfbot thinks this doesn't compile on Windows [1].  Looks like perhaps
a missing-#include problem?

regards, tom lane

[1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.81541




Re: BUG #15858: could not stat file - over 4GB

2020-02-28 Thread Juan José Santamaría Flecha
On Wed, Feb 5, 2020 at 12:47 PM Emil Iggland  wrote:

> The following review has been posted through the commitfest application:
> make installcheck-world:  not tested
> Implements feature:   tested, passed
> Spec compliant:   not tested
> Documentation:not tested
>

The latest version of this patch could benefit from an update. Please find
attached a new version.

Most changes are cosmetic, but they have been more extensive than a simple
rebase so I am changing the status back to 'needs review'.

To summarize those changes:
- Rename 'win32_stat.c' file to  'win32stat.c', as a better match of
project files.
- Improve indentation and comments.
- Remove cruft about old Windows versions.

Regards,

Juan José Santamaría Flecha


0001-Support-for-large-files-on-Win32-v5.patch
Description: Binary data


Re: BUG #15858: could not stat file - over 4GB

2020-02-05 Thread Emil Iggland
The following review has been posted through the commitfest application:
make installcheck-world:  not tested
Implements feature:   tested, passed
Spec compliant:   not tested
Documentation:not tested

I ran into this problem when using psql.exe and copy command. 

I have checked out 11.6-release tarball and applied the patch. 
The patch does not apply cleanly, but can be easily modified to apply. See Note 
1.
After applying the patch I built using "build psql" and ran the new psql.exe 
binary.

In order to test I have done the following: 
Against a PostgreSQL 11 server run two commands: 
"COPY public.table FROM 'C:/file'" and "\copy public.table FROM 'C:/file'"
The first one runs in the context of the server, and does not work. It aborts 
with an error saying "cannot stat file", as expected. 
The seconds on runs in the context of the new binary and does work. It copies 
data as expected. 



Note 1: 
src/tools/msvc/Mkvcbuild.pm should be 

- sprompt.c strerror.c tar.c thread.c getopt.c getopt_long.c dirent.c
- win32env.c win32error.c win32security.c win32setlocale.c);
+ sprompt.c tar.c thread.c getopt.c getopt_long.c dirent.c
+ win32env.c win32error.c win32security.c win32setlocale.c 
win32_stat.c);

The new status of this patch is: Waiting on Author


Re: BUG #15858: could not stat file - over 4GB

2019-10-28 Thread Juan José Santamaría Flecha
On Mon, Oct 28, 2019 at 3:29 PM william allen 
wrote:

> Hi - is this likely to be applied to an upcoming release? / How does a
> novice apply a patch..?
>
>
At this moment is missing review, so it is probably far from being
commitable. Any attention is appreciated and might help pushing it forward.
As a personal note, I have to check that is still applies before the
upcoming commitfest.

As for applying this patch you would need a Windows development
environment. I would recommend Visual Studio as a starting point [1]. You
also have a very visual guide in the wiki [2].

[1] https://www.postgresql.org/docs/current/install-windows.html
[2] https://wiki.postgresql.org/wiki/Working_With_VisualStudio

Regards,

Juan José Santamaría Flecha


RE: BUG #15858: could not stat file - over 4GB

2019-10-28 Thread william allen
Hi - is this likely to be applied to an upcoming release? / How does a novice 
apply a patch..?

Thanks

-Original Message-
From: Juan José Santamaría Flecha  
Sent: 04 September 2019 22:48
To: Tom Lane 
Cc: Michael Paquier ; williamedwinal...@live.com; 
pgsql-b...@lists.postgresql.org; Magnus Hagander ; 
PostgreSQL Hackers 
Subject: Re: BUG #15858: could not stat file - over 4GB

Thanks for looking into this.

On Fri, Aug 23, 2019 at 11:49 PM Tom Lane  wrote:
>
> Directly editing the configure script is Not Done ... or at least, 
> such changes wouldn't survive the next correctly-done configure 
> update.  You have to edit configure.in (or one of the sub-files in
> config/) and then regenerate configure using autoconf.
>
> It seems likely that we *don't* need or want this for Cygwin; that 
> should be providing a reasonable stat() emulation already.
> So probably you just want to add "AC_LIBOBJ(win32_stat)" to the stanza 
> beginning
>
> I'd also recommend that stat() fill all the fields in struct stat, 
> even if you don't have anything better to put there than zeroes.
> Otherwise you're just opening things up for random misbehavior.
>

Fixed.

> I'm not in a position to comment on the details of the conversion from 
> GetFileAttributesEx results to struct stat, but in general this seems 
> like a reasonable way to proceed.
>

Actually, due to the behaviour of GetFileAttributesEx with symbolic links I 
think that using GetFileInformationByHandle instead can give a more resilient 
solution. Also, by using a handle we get a good test for ERROR_DELETE_PENDING. 
This is the approach for the attached patch.

Regards,

Juan José Santamaría Flecha


Re: BUG #15858: could not stat file - over 4GB

2019-09-04 Thread Juan José Santamaría Flecha
Thanks for looking into this.

On Fri, Aug 23, 2019 at 11:49 PM Tom Lane  wrote:
>
> Directly editing the configure script is Not Done ... or at least,
> such changes wouldn't survive the next correctly-done configure
> update.  You have to edit configure.in (or one of the sub-files in
> config/) and then regenerate configure using autoconf.
>
> It seems likely that we *don't* need or want this for Cygwin;
> that should be providing a reasonable stat() emulation already.
> So probably you just want to add "AC_LIBOBJ(win32_stat)" to
> the stanza beginning
>
> I'd also recommend that stat() fill all the fields in struct stat,
> even if you don't have anything better to put there than zeroes.
> Otherwise you're just opening things up for random misbehavior.
>

Fixed.

> I'm not in a position to comment on the details of the conversion from
> GetFileAttributesEx results to struct stat, but in general this
> seems like a reasonable way to proceed.
>

Actually, due to the behaviour of GetFileAttributesEx with symbolic
links I think that using GetFileInformationByHandle instead can give a
more resilient solution. Also, by using a handle we get a good test
for ERROR_DELETE_PENDING. This is the approach for the attached patch.

Regards,

Juan José Santamaría Flecha


0001-WIP-support-for-large-files-on-Win32-v4.patch
Description: Binary data