Re: Changing encoding of a file : What should happen to CRLF in file ?

[Ashish Negi]

Great work !!!
Thanks

On Fri, Nov 24, 2017 at 1:55 AM, Torsten Bögershausen  wrote:
> On Thu, Nov 23, 2017 at 10:01:40PM +0530, Ashish Negi wrote:
>> Thanks for confirming.
>> Is it possible to track this via a bug number ?
>> It will help me to try out the fix when its available.
>>
>
> No worry, the fix is nearly complete and will come out in a couple of days.

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Torsten Bögershausen]

On Thu, Nov 23, 2017 at 10:01:40PM +0530, Ashish Negi wrote:
> Thanks for confirming.
> Is it possible to track this via a bug number ?
> It will help me to try out the fix when its available.
> 

No worry, the fix is nearly complete and will come out in a couple of days.

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Ashish Negi]

Thanks for confirming.
Is it possible to track this via a bug number ?
It will help me to try out the fix when its available.


On Thu, Nov 16, 2017 at 9:45 PM, Torsten Bögershausen  wrote:
> On Thu, Nov 16, 2017 at 12:35:33AM +0530, Ashish Negi wrote:
>> On windows :
>> > git --version
>> git version 2.14.2.windows.2
>>
>> On linux :
>> > git --version
>> git version 2.7.4
>>
>> I would like to understand the solution :
>> If i understood it correctly : it removes file_name.txt from index, so
>> git forgets about it.
>> we then add the file again after changing encoding. This time, git
>> takes it as utf-8 file and converts crlf to lf when storing it
>> internally.
>> Right ?
>
> Yes, exactly.
> (In a coming release of Git there will be a "git add --renormalize 
> " )
>
>>
>> Thank you for the support.
>>
>
> Thanks for a clean bug report.
> Actually it is a bug, I put it on my to do list
>
>

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Torsten Bögershausen]

On Thu, Nov 16, 2017 at 12:35:33AM +0530, Ashish Negi wrote:
> On windows :
> > git --version
> git version 2.14.2.windows.2
> 
> On linux :
> > git --version
> git version 2.7.4
> 
> I would like to understand the solution :
> If i understood it correctly : it removes file_name.txt from index, so
> git forgets about it.
> we then add the file again after changing encoding. This time, git
> takes it as utf-8 file and converts crlf to lf when storing it
> internally.
> Right ?

Yes, exactly.
(In a coming release of Git there will be a "git add --renormalize " )

> 
> Thank you for the support.
> 

Thanks for a clean bug report.
Actually it is a bug, I put it on my to do list

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Ashish Negi]

On windows :
> git --version
git version 2.14.2.windows.2

On linux :
> git --version
git version 2.7.4

I would like to understand the solution :
If i understood it correctly : it removes file_name.txt from index, so
git forgets about it.
we then add the file again after changing encoding. This time, git
takes it as utf-8 file and converts crlf to lf when storing it
internally.
Right ?

Thank you for the support.


On Wed, Nov 15, 2017 at 10:42 PM, Torsten Bögershausen  wrote:
> On Wed, Nov 15, 2017 at 01:41:42PM +0530, Ashish Negi wrote:
>> > If you commit the file, it will be stored with LF in the index,
>> This is what i believe is not happening.
>>
>> Lets do this with a public repository and steps which are reproducible.
>> I have created a repo : https://github.com/ashishnegi/git_encoding
>>
>> If you clone this repo in linux and run `git status`, you will find
>> that file is modified.
>
> This is what what I get:
>
>  git ls-files --eol
> i/lfw/lfattr/text=auto  .gitattributes
> i/crlf  w/crlf  attr/text=auto  file_name.txt
>
> (And you get the same, I think)
> Newer versions of Git (>=2.10) will -not- treat file_name.txt as changed,
> older versions do.
> What does "git --version" say on your Linux machine ?
>
> However, if you want to fix it, you want to either end up with
>
> A)
> git ls-files --eol
> i/lfw/lfattr/text=auto  .gitattributes
> i/lfw/crlf  attr/text=auto  file_name.txt
>
> or
> B)
> git ls-files --eol
> i/lfw/lfattr/text=auto  .gitattributes
> i/crlf  w/crlf  attr/-text  file_name.txt
>
> (The "attr/-text" means "don't change the line endings")
>
> Both A) or B) will work, typically A) is preferred.
>
> You should be able to achive A) :
>  git rm --cached file_name.txt  && git add file_name.txt
> rm 'file_name.txt'
> warning: CRLF will be replaced by LF in file_name.txt.
> The file will have its original line endings in your working directory.
>
> git ls-files --eol
> i/lfw/lfattr/text=auto  .gitattributes
> i/lfw/crlf  attr/text=auto  file_name.txt
>
> [snip the rest]

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Torsten Bögershausen]

On Wed, Nov 15, 2017 at 01:41:42PM +0530, Ashish Negi wrote:
> > If you commit the file, it will be stored with LF in the index,
> This is what i believe is not happening.
> 
> Lets do this with a public repository and steps which are reproducible.
> I have created a repo : https://github.com/ashishnegi/git_encoding
> 
> If you clone this repo in linux and run `git status`, you will find
> that file is modified.

This is what what I get:

 git ls-files --eol
i/lfw/lfattr/text=auto  .gitattributes
i/crlf  w/crlf  attr/text=auto  file_name.txt

(And you get the same, I think)
Newer versions of Git (>=2.10) will -not- treat file_name.txt as changed,
older versions do.
What does "git --version" say on your Linux machine ?

However, if you want to fix it, you want to either end up with

A)
git ls-files --eol
i/lfw/lfattr/text=auto  .gitattributes
i/lfw/crlf  attr/text=auto  file_name.txt

or
B)
git ls-files --eol
i/lfw/lfattr/text=auto  .gitattributes
i/crlf  w/crlf  attr/-text  file_name.txt

(The "attr/-text" means "don't change the line endings")

Both A) or B) will work, typically A) is preferred.

You should be able to achive A) :
 git rm --cached file_name.txt  && git add file_name.txt 
rm 'file_name.txt'
warning: CRLF will be replaced by LF in file_name.txt.
The file will have its original line endings in your working directory.

git ls-files --eol
i/lfw/lfattr/text=auto  .gitattributes
i/lfw/crlf  attr/text=auto  file_name.txt

[snip the rest]

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Ashish Negi]

> If you commit the file, it will be stored with LF in the index,
This is what i believe is not happening.

Lets do this with a public repository and steps which are reproducible.
I have created a repo : https://github.com/ashishnegi/git_encoding

If you clone this repo in linux and run `git status`, you will find
that file is modified.

About repo :
Repo have 2 commits, done on windows machine.
First one check in a utf-16le encoded file which has crlf. crlf will
not be converted to lf in index as git treats it as binary file.
2nd commit changes encoding to utf-8 and commits.
This commit does not change crlf to lf in index, even though new
format is utf-8 which is text based for git. This is the crux of
problem.

I have attached all commands i ran on windows while creating the repo.
I tried to capture all information that i could give.
Please have a look. It might be useful.

Finally, thank you Torsten for giving your time to the problem. Really
appreciate it.

On Tue, Nov 14, 2017 at 10:39 PM, Torsten Bögershausen  wrote:
> (Back to the beginning)
>
> You have a file ApplicationManifest.xml
> It is encoded in UTF-16 (and has CRLF)
>
> You convert it into UTF-8
> The file has still CRLF (in the worktree)
>
> Now you add it and make a commit.
> Under both Linux and Windows you have "text=auto".
>
> I assume that you have efficiently core.eol=lf under Linux
> and core.eol=crlf on Windows.
>
> (That is the default, when you don't change anything)
>
> Now, what happens to the CRLF?
> If you commit the file, it will be stored with LF in the index,
> on both systems.
> On checkout, Windows will convert them into CRLF, but Linux will not.
>
> That why you see
>>On linux, during committing i get warning : warning: CRLF will be
>>replaced by LF in …file_name..
>
> All in all there is nothing wrong, at least as I see it.
>
> The question remains:
> Do you need CRLF in Linux ?
> Probably not, but if yes, plase add a line
>
> *.xml text eol=crlf
>
> to your
> .gitattributes
>
> Otherwise your .gitconfig looks good to  me.
>
>
>
>
>
>
git_encoding_repo>git config --global core.safecrlf
true

git_encoding_repo>git config  core.autocrlf
false

git_encoding_repo>git config --get core.autocrlf
false

git_encoding_repo>cat .gitattributes
# Set the default behavior, in case people don't have core.autocrlf set.
* text=auto

*.vcxproj eol=crlf
*.sh  eol=lf

# Denote all files that are truly binary and should not be modified.
*.exe binary
*.dll binary
*.pdb binary
*.ico binary
*.png binary
*.jpg binary

git_encoding_repo>git status
On branch master

No commits yet

Untracked files:
  (use "git add ..." to include in what will be committed)

.gitattributes
file_name.txt

nothing added to commit but untracked files present (use "git add" to track)

git_encoding_repo>git ls-files --eol file_name.txt

git_encoding_repo>git add .

git_encoding_repo>git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached ..." to unstage)

new file:   .gitattributes
new file:   file_name.txt


git_encoding_repo>git ls-files --eol file_name.txt
i/-text w/-text attr/text=auto  file_name.txt

git_encoding_repo>git commit -m "Commit utf-16le encoded file which has crlf."
[master (root-commit) 91fe3bd] Commit utf-16le encoded file which has crlf.
 2 files changed, 13 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 file_name.txt

## At this time, i changed file encoding to utf-8.

git_encoding_repo>git status
On branch master
nothing to commit, working tree clean

git_encoding_repo>git add -p
Only binary files changed.

git_encoding_repo>git status
On branch master
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)

modified:   file_name.txt

no changes added to commit (use "git add" and/or "git commit -a")

git_encoding_repo>git add file_name.txt

git_encoding_repo>git status
On branch master
Changes to be committed:
  (use "git reset HEAD ..." to unstage)

modified:   file_name.txt


git_encoding_repo>git commit -m "Change encoding of file to utf-8"
[master 179c27b] Change encoding of file to utf-8
 1 file changed, 0 insertions(+), 0 deletions(-)
 rewrite file_name.txt (100%)

git_encoding_repo>git remote add origin 
https://github.com/ashishnegi/git_encoding.git

git_encoding_repo>git push -u origin master
Counting objects: 7, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 837 bytes | 837.00 KiB/s, done.
Total 7 (delta 0), reused 0 (delta 0)
To https://github.com/ashishnegi/git_encoding.git
 * [new branch]  master -> master
Branch master set up to track remote branch master from origin.

git_encoding_repo>git ls-files --eol file_name.txt
i/crlf  w/crlf  attr/text=auto  file_name.txt

git_encoding_repo>

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Torsten Bögershausen]

(Back to the beginning)

You have a file ApplicationManifest.xml
It is encoded in UTF-16 (and has CRLF)

You convert it into UTF-8
The file has still CRLF (in the worktree)

Now you add it and make a commit.
Under both Linux and Windows you have "text=auto".

I assume that you have efficiently core.eol=lf under Linux
and core.eol=crlf on Windows.

(That is the default, when you don't change anything)

Now, what happens to the CRLF?
If you commit the file, it will be stored with LF in the index,
on both systems.
On checkout, Windows will convert them into CRLF, but Linux will not.

That why you see
>On linux, during committing i get warning : warning: CRLF will be
>replaced by LF in …file_name..

All in all there is nothing wrong, at least as I see it.

The question remains:
Do you need CRLF in Linux ?
Probably not, but if yes, plase add a line

*.xml text eol=crlf

to your
.gitattributes

Otherwise your .gitconfig looks good to  me.

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Torsten Bögershausen]

On 2017-11-14 17:13, Ashish Negi wrote:
> Running the command gives me :
> 
>   git ls-files --eol file_name
>   i/-text w/-text attr/text=auto  file_name
> 

That is strange to me:
According to that, Git would treat the file as text=auto.
And the content is "not next", so there is no need to convert.
Do you have configured any filters ?

Is this a public repo ?
if not: Is there any chance that you provide a public example,
so that we can have a look ?

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Ashish Negi]

After changing the encoding of file to utf-8, same command gives:

git ls-files --eol file_name
i/lfw/crlf  attr/text=auto  ApplicationManifest.xml

On Tue, Nov 14, 2017 at 9:43 PM, Ashish Negi  wrote:
> Running the command gives me :
>
>   git ls-files --eol file_name
>   i/-text w/-text attr/text=auto  file_name

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Ashish Negi]

Running the command gives me :

  git ls-files --eol file_name
  i/-text w/-text attr/text=auto  file_name

Re: Changing encoding of a file : What should happen to CRLF in file ?

[Torsten Bögershausen]

On 2017-11-14 13:31, Ashish Negi wrote:
> Hello
> 
> I have a cross platform project. I have a utf-16 file in it.
> I changed its encoding to urf-8 and committed. When i pulled the file
> in Linux, it shows that file is modified. This means that the commit
> which changed the encoding does not convert crlf to lf, when new
> format is text based (utf-8).
> 
> Steps to reproduce:
> 
>In windows :
> 
> Change encoding of file from utf-16 to utf-8.
> Commit the change.
> 
>In linux:
> 
> Pull your branch.
> You will see the issue of file being shown as modified even though
> you have not used it.
> 
> 
> If i change the file encoding in linux and commit. Then if i do git
> pull in windows, i don't see the issue.
> In linux, during committing i get warning : warning: CRLF will be
> replaced by LF in …file_name..
> 
> Here are my configuration :
> 
> 
>> git config --global --get core.autocrlf
> 
> false
> 
> 
>> git config  --get core.autocrlf
> 
> false
> 
> 
> 
>> cat E:\work\WindowsFabric\.gitattributes
> 
> 
> # Set the default behavior, in case people don't have core.autocrlf set.
> 
> * text=auto
> *.vcxproj eol=crlf
> *.sh  eol=lf
> 
> # Denote all files that are truly binary and should not be modified.
> *.exe binary
> *.dll binary
> *.pdb binary
> *.ico binary
> *.png binary
> *.jpg binary
> 
> 
>> git --version
> git version 2.14.2.windows.2
> 
> 
> I played around with core.autocrlf values like true and native, but
> that didn't help.
> 
> The behavior is inconsistent across platforms, and windows version is
> giving me problem.
> 
> Can someone suggest right settings for this ? Or is this a bug in Git.
> 

I don't think it is a bug :-)
What does
git ls-files --eol …file_name
give you under Windows ?