Proposal

[Ms Melisa Mehmet]

-- 
Hello 

i want to invest in your region i have some funds under my management
please let me know if we can work together on this investment plan.

Regards 
Melisa Mehmet

Greetings in the name of God, Business proposal in God we trust

[Mrs, Suran Yoda]

Greetings in the name of God

Dear Friend


Greetings in the name of God,please let this not sound strange to you
for my only surviving lawyer who would have done this died early this
year.I prayed and got your email id from your country guestbook. I am
Mrs Suran Yoda from London,I am 72 years old,i am suffering from a
long time cancer of the lungs which also affected my brain,from all
indication my conditions is really deteriorating and it is quite
obvious that,according to my doctors they have advised me that i may
not live for the next two months,this is because the cancer stage has
gotten to a very bad stage.I am married to (Dr Andrews Yoda) who
worked with the Embassy of United Kingdom in South

Africa for nine years,Before he died in 2004.

I was bred up from a motherless babies home and was married to my late
husband for Thirty years without a child,my husband died in a fatal
motor accident Before his death we were true believers.Since his death
I decided not to re-marry,I sold all my inherited belongings and
deposited all the sum of $6.5 Million dollars with Bank in South
Africa.Though what disturbs me mostly is the cancer. Having known my
condition I decided to donate this fund to church,i want you as God
fearing person,to also use this money to fund church,orphanages and
widows,I took this decision,before i rest in peace because my time
will so on be up.

The Bible made us to understand that blessed are the hands that
giveth. I took this decision because I don`t have any child that will
inherit this money and my husband's relatives are not Christians and I
don`t want my husband hard earned money to be misused by unbelievers.
I don`t want a situation where these money will be used in an ungodly
manner,hence the reason for taking this bold decision.I am not afraid
of death hence i know where am going.Presently,I'm with my laptop in a
hospital here in London where I have been undergoing treatment for
cancer of the lungs.

As soon as I receive your reply I shall give you the contact of the
Bank.I will also issue you a letter of authority that will prove you
as the new beneficiary of my fund.Please assure me that you will act
accordingly as I stated.Hoping to hear from you soon.

Remain blessed in the name of the Lord.

Yours in Christ,
Mrs Suran Yoda

Business Proposal.

[Douglas Flint]

Dear Friend,

I am interested in establishing and operating a very viable business as a means 
of investment abroad. I do not know too well on how this is done in your 
country, So I will need you to help me in this regard.

My preference is any good profit yielding business and I would appreciate any 
viable ideas you could come up with. I will also need you to help me look for 
properties like homes and lands for sale as I am proposing to invest the sum of 
Ten Million Great British Pounds (10,000,000 GBP) for this. I do not know if 
you can and will be of help to me.

For a brief on my personality; 

My need for this business proposition and to acquire these properties is very 
urgent as I am planing to retire from my position as the newly retired chairman 
of HSBC bank, I did stepped down on the 30th/September/2017.
As business inclined person, I want you to come up with your individual 
business/ investment ideas as I will welcome it from the bottom of my heart.


Please I expect your good and prompt reply so that we can proceed swiftly on 
how the funds can be sent to you as the beneficiary. 

Best Regards,
Douglas J Flint.

Greetings in the name of God, Business proposal in God we trust

[Mrs, Suran Yoda]

Hello Dear,

   I am DR DANIEL MMINELE THE DEPUTY GOVERNOR OF SOUTH AFRICAN RESERVE
BANK,YOU CAN VISIT THIS WEB SITE FOR YOUR
CONFIRMATION:http://www.whoswhosa.co.za/south-african-reserve-bank-20991
,

  I know my message will come to you as a surprise. Don't worry I was
totally convinced to write you, I hoped that you will not expose or
betray this trust and confident that i am about to repose on you for
the mutual benefit of our families.
I need your urgent assistance in transferring sum of US$10.200,000
(Ten Million two hundred thousand Dollars US). into your account
within 10 to 14 banking days. This requires a private arrangement; the
details of the transaction will be furnished to you if you indicate
your interest in this proposal.

This money has been dormant for years in our bank without claim, I
want the bank to release the money to you as the nearest person (owner
of the account)who died along with his entire family during the Iraq
war in 2006

I don't want the money to go into our bank treasurer account as an
abandoned fund. So this is the reason why i contact you so that the
bank will release the money to you as the next of kin to the death
customer and we share the money together. Please i would like you to
keep this proposal as a top secret.

Upon receipt of your reply, I will give you full details on how the
business project will be executed and also note that you will have 40%
of the above mentioned sum.

Kindly fill this information's requested below in returns then i will
give you more details with application form for the claim.

reply at (danielmmine...@yahoo.com)

Your full name: 
Your Country: 
Phone Number: 
Your Age and occupation: 

I expect your urgent response if you agree to handle this project.

Best regard.

Dr, Daniel Mminele the deputy governor of south African reserve bank

Greetings in the name of God, Business proposal in God we trust

[Mrs, Suran Yoda]

Greetings in the name of God

Dear Friend


Greetings in the name of God,please let this not sound strange to you
for my only surviving lawyer who would have done this died early this
year.I prayed and got your email id from your country guestbook. I am
Mrs Suran Yoda from London,I am 72 years old,i am suffering from a
long time cancer of the lungs which also affected my brain,from all
indication my conditions is really deteriorating and it is quite
obvious that,according to my doctors they have advised me that i may
not live for the next two months,this is because the cancer stage has
gotten to a very bad stage.I am married to (Dr Andrews Yoda) who
worked with the Embassy of United Kingdom in South

Africa for nine years,Before he died in 2004.

I was bred up from a motherless babies home and was married to my late
husband for Thirty years without a child,my husband died in a fatal
motor accident Before his death we were true believers.Since his death
I decided not to re-marry,I sold all my inherited belongings and
deposited all the sum of $6.5 Million dollars with Bank in South
Africa.Though what disturbs me mostly is the cancer. Having known my
condition I decided to donate this fund to church,i want you as God
fearing person,to also use this money to fund church,orphanages and
widows,I took this decision,before i rest in peace because my time
will so on be up.

The Bible made us to understand that blessed are the hands that
giveth. I took this decision because I don`t have any child that will
inherit this money and my husband's relatives are not Christians and I
don`t want my husband hard earned money to be misused by unbelievers.
I don`t want a situation where these money will be used in an ungodly
manner,hence the reason for taking this bold decision.I am not afraid
of death hence i know where am going.Presently,I'm with my laptop in a
hospital here in London where I have been undergoing treatment for
cancer of the lungs.

As soon as I receive your reply I shall give you the contact of the
Bank.I will also issue you a letter of authority that will prove you
as the new beneficiary of my fund.Please assure me that you will act
accordingly as I stated.Hoping to hear from you soon.

Remain blessed in the name of the Lord.

Yours in Christ,
Mrs Suran Yoda

Project Proposal

[Joseph Taylorl]

Dear sir,

I am Barrister Joseph Taylor, a legal Solicitor. I was the Personal
Attorney and legal adviser to Mr. John ALBIN, a national of your country,
who was an expatriate engineer to British Petroleum oil Company. My
client, his wife, and their three children were involved in the ill fated
Kenya Airways crash in the coasts of Abidjan in January 2000 in which all
passengers on board died. Since then I have made several inquiries to your
embassy to locate any of my clients extended relatives, this has proved
unsuccessful.

After these several unsuccessful attempts, I decided to trace his
relatives over the Internet, to locate any member of his family but of no
avail, hence I contacted you. I have contacted you to assist in
repatriating the fund and property left behind by my client before the
bank diverts the fund to government treasury.

I seek your consent to present you as a relative to the deceased since you
are from the same country so that his fund valued at Twenty Million United
States Dollars (US$20,000,000.00) can be transferred to you by the bank. I
will procure all the necessary claim documents from the court to make your
claim most legal and legitimate. I will also present you officially to the
bank since I am in a position to do so being the executor of his will.

We shall decide on the sharing formula once I hear from you. However, I
will expect to hear from you urgently because time is running out on us to
claim the fund.

I am Waiting for your urgent response.

Best Regards,
Joseph Taylor

BUSINESS PROPOSAL

[LING LUNG]

Please I   like you to keep this proposal as a top secret and delete it if you 
are not interested and get back to 
me if you are interested for  details as regards to the transfer of $24,500,000 
to you. This money initially 
belongs to a Libyan client who died in the libya crisis  and had no next of kin 
in his account-opening 
package in my bank here in Hong kong where I am a bank director. In other to 
achieve this, I shall 
require your full name, and telephone number to reach you.Most importantly,  a 
confirmation of 
acceptance from you will be needed after which I shall furnish you with the 
full details of this transaction.

Kindly reply to linglung0...@gmail.com

Respectfully,

Ling Lung

BUSINESS PROPOSAL

[LING LUNG]

Please I   like you to keep this proposal as a top secret and delete it if you 
are not interested and get back to 
me if you are interested for  details as regards to the transfer of $24,500,000 
to you. This money initially 
belongs to a Libyan client who died in the libya crisis  and had no next of kin 
in his account-opening 
package in my bank here in Hong kong where I am a bank director. In other to 
achieve this, I shall 
require your full name, and telephone number to reach you.Most importantly,  a 
confirmation of 
acceptance from you will be needed after which I shall furnish you with the 
full details of this transaction.

Kindly reply to linglung0...@gmail.com

Respectfully,

Ling Lung

BUSINESS PROPOSAL

[LING LUNG]

Please I   like you to keep this proposal as a top secret and delete it if you 
are not interested and get back to 
me if you are interested for  details as regards to the transfer of $24,500,000 
to you. This money initially 
belongs to a Libyan client who died in the libya crisis  and had no next of kin 
in his account-opening 
package in my bank here in Hong kong where I am a bank director. In other to 
achieve this, I shall 
require your full name, and telephone number to reach you.Most importantly,  a 
confirmation of 
acceptance from you will be needed after which I shall furnish you with the 
full details of this transaction.

Kindly reply to linglung0...@gmail.com

Respectfully,

Ling Lung

BUSINESS PROPOSAL

[LING LUNG]

Please I   like you to keep this proposal as a top secret and delete it if you 
are not interested and get back to 
me if you are interested for  details as regards to the transfer of $24,500,000 
to you. This money initially 
belongs to a Libyan client who died in the libya crisis  and had no next of kin 
in his account-opening 
package in my bank here in Hong kong where I am a bank director. In other to 
achieve this, I shall 
require your full name, and telephone number to reach you.Most importantly,  a 
confirmation of 
acceptance from you will be needed after which I shall furnish you with the 
full details of this transaction.

Kindly reply to linglung0...@gmail.com

Respectfully,

Ling Lung

Re: fetch-any-blob / ref-in-want proposal

[Jonathan Tan]

On Mon, Jul 24, 2017 at 12:38 PM, Orgad Shaneh  wrote:
> Sorry, I thought it's the same thing. I mean ref-in-want[1]. This
> issue in gerrit[2] was closed, claiming that ref-in-want will solve
> it. I just want to know if this is likely to be merged soon enough (I
> consider several months "soon enough"), or should I look for other
> solutions.
>
> - Orgad
>
> [1] 
> https://public-inbox.org/git/cover.1485381677.git.jonathanta...@google.com/
> [2] https://bugs.chromium.org/p/gerrit/issues/detail?id=175#c24

Right now, we have been working on other things - so it is unlikely
that it will be merged soon enough.

Re: fetch-any-blob / ref-in-want proposal

[Orgad Shaneh]

On Mon, Jul 24, 2017 at 7:13 PM, Jonathan Tan  wrote:
> On Sun, 23 Jul 2017 09:41:50 +0300
> Orgad Shaneh  wrote:
>
>> Hi,
>>
>> Jonathan Tan proposed a design and a patch series for requesting a
>> specific ref on fetch 4 months ago[1].
>>
>> Is there any progress with this?
>>
>> - Orgad
>>
>> [1] 
>> https://public-inbox.org/git/ffd92ad9-39fe-c76b-178d-6e3d6a425...@google.com/
>
> Do you mean requesting a specific blob (as referenced by your link)? If yes, 
> it is still being discussed. One such discussion is here: [1]
>
> If you mean ref-in-want, I don't recall anything being done since then.
>
> [1] 
> https://public-inbox.org/git/cover.1499800530.git.jonathanta...@google.com/

Sorry, I thought it's the same thing. I mean ref-in-want[1]. This
issue in gerrit[2] was closed, claiming that ref-in-want will solve
it. I just want to know if this is likely to be merged soon enough (I
consider several months "soon enough"), or should I look for other
solutions.

- Orgad

[1] https://public-inbox.org/git/cover.1485381677.git.jonathanta...@google.com/
[2] https://bugs.chromium.org/p/gerrit/issues/detail?id=175#c24

Re: fetch-any-blob / ref-in-want proposal

[Jonathan Tan]

On Sun, 23 Jul 2017 09:41:50 +0300
Orgad Shaneh  wrote:

> Hi,
> 
> Jonathan Tan proposed a design and a patch series for requesting a
> specific ref on fetch 4 months ago[1].
> 
> Is there any progress with this?
> 
> - Orgad
> 
> [1] 
> https://public-inbox.org/git/ffd92ad9-39fe-c76b-178d-6e3d6a425...@google.com/

Do you mean requesting a specific blob (as referenced by your link)? If yes, it 
is still being discussed. One such discussion is here: [1]

If you mean ref-in-want, I don't recall anything being done since then.

[1] https://public-inbox.org/git/cover.1499800530.git.jonathanta...@google.com/

fetch-any-blob / ref-in-want proposal

[Orgad Shaneh]

Hi,

Jonathan Tan proposed a design and a patch series for requesting a
specific ref on fetch 4 months ago[1].

Is there any progress with this?

- Orgad

[1] 
https://public-inbox.org/git/ffd92ad9-39fe-c76b-178d-6e3d6a425...@google.com/

Re: Proposal

[Elodie Azeez]

Hello,

Compliment of the day to you. I am Mrs. Elodie, I am sending this brief
letter to solicit your partnership to transfer $5.5 million US Dollars into
your account for investment in your country. I shall send you more information
and procedures when I receive positive response from you.

Thank you.
Mrs. Elodie Azeez

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Junio C Hamano]

Marc Branchaud  writes:

> OTOH, we shouldn't need to do any conflict resolution on these
> "tracking" refs:  The remote side should update the refs
> properly. Nobody should make local changes to these refs.
>
> I guess I'm advocating that git should only allow "tracking" refs to
> be updated by a fetch, or a successful push of a local, non-"tracking"
> ref.

I do not think anybody is imagining new things to happen inside
refs/tracking other than by fetching, just like refs/remotes/ refs
behave like so.

What I was discussing was mostly the step next to the introduction
of tracking/.  Some things are useful merely by existing (e.g. tags
you got from remote, as long as you can easily refer to them, are
useful to you).  Some other things are cumbersome to use until you
manage to consolidate your views with theirs in order to make
collective progress, and they require merging (e.g. notes).

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Marc Branchaud]

On 2017-06-23 04:54 PM, Junio C Hamano wrote:

Jacob Keller  writes:


Instead, lets add support for a new refs/tracking/* hierarchy which is
laid out in such a way to avoid this inconsistency. All refs in
"refs/tracking//*" will include the complete ref, such that
dropping the "tracking/" part will give the exact ref name as it
is found in the upstream. Thus, we can track any ref here by simply
fetching it into refs/tracking//*.


I do think this overall is a good goal to aim for wrt "tracking",
i.e.  keeping a pristine copy of what we observed from the outside
world.  In addition to what you listed as "undecided" below,
however, I expect that this will highlight a lot of trouble in
"working together", i.e. reconciling the differences of various
world views and moving the project forward, in two orthogonal axes:

  - Things pointed at by some refs (e.g. notes/, but I think the
".gitmodules equivalent that is not tied to any particular commit
in the superproject" Jonathan Nieder and friends advocate falls
into the same category) do not correspond to the working tree
contents, and merging their contents will always pose challenge
when we need to prepare for conflict resolution.


OTOH, we shouldn't need to do any conflict resolution on these 
"tracking" refs:  The remote side should update the refs properly. 
Nobody should make local changes to these refs.


I guess I'm advocating that git should only allow "tracking" refs to be 
updated by a fetch, or a successful push of a local, non-"tracking" ref.


M.



  - Things pointed at by some other refs (e.g. tags/) do not make
sense to be merged.  You may locally call a commit with a tag
"wip", while your friends may use the same "wip" tag to point at
a different one.  Both are valid, and more importantly, there is
no point even trying to reconcile what the "wip" tag means in the
project globally.

For the former, I expect that merging these non-working tree
contents will all have to require some specific tool that is
tailored for the meaning each hierarchy has, just like "git notes
merge" gives a solution that is very specific to the notes refs (I
do not know how well "notes merge" works in practice, though).

For the latter, having a separate tracking hierarchy is a strict
improvement from "tracking" point of view, but I think "working
together" also needs a well designed set of new look-up rules, and a
new convention.  For example, some tags may not want to be shared
(e.g. "wip" example above) even though they should be easy to access
by those who already have them (e.g. "git log ..wip" should work
without having to say "git log ..refs/local-tags/wip", even if we
decide to implement the "some tags may not want to be shared"
segregation by introducing a new hierarchy).  And also a shared tag
that is copied to "refs/tracking/origin/tags/v1.0" would need a way
better than "git log tracking/origin/tags/v1.0" for this mechanism
to be useful in everyday workflow.

Thanks.

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Jacob Keller]

On Fri, Jun 23, 2017 at 1:54 PM, Junio C Hamano <gits...@pobox.com> wrote:
> Jacob Keller <jacob.e.kel...@intel.com> writes:
>
>> Instead, lets add support for a new refs/tracking/* hierarchy which is
>> laid out in such a way to avoid this inconsistency. All refs in
>> "refs/tracking//*" will include the complete ref, such that
>> dropping the "tracking/" part will give the exact ref name as it
>> is found in the upstream. Thus, we can track any ref here by simply
>> fetching it into refs/tracking//*.
>
> I do think this overall is a good goal to aim for wrt "tracking",
> i.e.  keeping a pristine copy of what we observed from the outside
> world.  In addition to what you listed as "undecided" below,
> however, I expect that this will highlight a lot of trouble in
> "working together", i.e. reconciling the differences of various
> world views and moving the project forward, in two orthogonal axes:
>

I agree, I think this is definitely a problem that we'd want/have to solve.

>  - Things pointed at by some refs (e.g. notes/, but I think the
>".gitmodules equivalent that is not tied to any particular commit
>in the superproject" Jonathan Nieder and friends advocate falls
>into the same category) do not correspond to the working tree
>contents, and merging their contents will always pose challenge
>when we need to prepare for conflict resolution.

This is part of why I started by looking at notes, since there is
already some code for notes merge, it just doesn't have a standard
place to store "remote" notes. I've wanted that ability to enable more
ease of sharing.

>
>  - Things pointed at by some other refs (e.g. tags/) do not make
>sense to be merged.  You may locally call a commit with a tag
>"wip", while your friends may use the same "wip" tag to point at
>a different one.  Both are valid, and more importantly, there is
>no point even trying to reconcile what the "wip" tag means in the
>project globally.
>
> For the former, I expect that merging these non-working tree
> contents will all have to require some specific tool that is
> tailored for the meaning each hierarchy has, just like "git notes
> merge" gives a solution that is very specific to the notes refs (I
> do not know how well "notes merge" works in practice, though).

Agreed. That's also part of why I'm looking at doing it as a partial
"one hierarchy at a time" proposal, I've got a somewhat better format
for that coming Soon(TM)

>
> For the latter, having a separate tracking hierarchy is a strict
> improvement from "tracking" point of view, but I think "working
> together" also needs a well designed set of new look-up rules, and a
> new convention.  For example, some tags may not want to be shared
> (e.g. "wip" example above) even though they should be easy to access
> by those who already have them (e.g. "git log ..wip" should work
> without having to say "git log ..refs/local-tags/wip", even if we
> decide to implement the "some tags may not want to be shared"
> segregation by introducing a new hierarchy).  And also a shared tag
> that is copied to "refs/tracking/origin/tags/v1.0" would need a way
> better than "git log tracking/origin/tags/v1.0" for this mechanism
> to be useful in everyday workflow.
>
> Thanks.
>

My thoughts on tags are sort of the following:

keep the default semantics of tags get copied into our local
refs/tags, (if we're fetching them). However, create some mechanism
for easily showing when you have a local tag with the same name as a
remote tag which don't point at the same thing, by using the
refs/tracking. Possibly, any time you grab a tag, check and see if the
remote version is different and "do something" like complain, or show
both, or give some advice (possibly with a new tool to "pull in the
remote tag if you verify it's accurate, etc".

This way, you generally work with tags the same way but get added
protection for knowing when your local tags differ from what remotes
show?

Thanks,
Jake

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Junio C Hamano]

Jacob Keller  writes:

> Instead, lets add support for a new refs/tracking/* hierarchy which is
> laid out in such a way to avoid this inconsistency. All refs in
> "refs/tracking//*" will include the complete ref, such that
> dropping the "tracking/" part will give the exact ref name as it
> is found in the upstream. Thus, we can track any ref here by simply
> fetching it into refs/tracking//*.

I do think this overall is a good goal to aim for wrt "tracking",
i.e.  keeping a pristine copy of what we observed from the outside
world.  In addition to what you listed as "undecided" below,
however, I expect that this will highlight a lot of trouble in
"working together", i.e. reconciling the differences of various
world views and moving the project forward, in two orthogonal axes:

 - Things pointed at by some refs (e.g. notes/, but I think the
   ".gitmodules equivalent that is not tied to any particular commit
   in the superproject" Jonathan Nieder and friends advocate falls
   into the same category) do not correspond to the working tree
   contents, and merging their contents will always pose challenge
   when we need to prepare for conflict resolution.

 - Things pointed at by some other refs (e.g. tags/) do not make
   sense to be merged.  You may locally call a commit with a tag
   "wip", while your friends may use the same "wip" tag to point at
   a different one.  Both are valid, and more importantly, there is
   no point even trying to reconcile what the "wip" tag means in the
   project globally.

For the former, I expect that merging these non-working tree
contents will all have to require some specific tool that is
tailored for the meaning each hierarchy has, just like "git notes
merge" gives a solution that is very specific to the notes refs (I
do not know how well "notes merge" works in practice, though).

For the latter, having a separate tracking hierarchy is a strict
improvement from "tracking" point of view, but I think "working
together" also needs a well designed set of new look-up rules, and a
new convention.  For example, some tags may not want to be shared
(e.g. "wip" example above) even though they should be easy to access
by those who already have them (e.g. "git log ..wip" should work
without having to say "git log ..refs/local-tags/wip", even if we
decide to implement the "some tags may not want to be shared"
segregation by introducing a new hierarchy).  And also a shared tag
that is copied to "refs/tracking/origin/tags/v1.0" would need a way
better than "git log tracking/origin/tags/v1.0" for this mechanism
to be useful in everyday workflow.

Thanks.

Re: [RFC PATCH] proposal for refs/tracking/* hierarchy

[Jacob Keller]

On Fri, Jun 23, 2017 at 6:52 AM, Jacob Keller  wrote:
> From: Jacob Keller 
>
> Historically, git has tracked the status of remote branches (heads) in
> refs/remotes//*. This is necessary and useful as it allows users
> to track the difference between their local work and the last known
> status of the remote work.
>
> Unfortunately this hierarchy is limited in that it only tracks branches.
> Additionally, it is not feasible to extend it directly, because the
> layout would become ambiguous. For example, if a user happened to have
> a local branch named "notes" it could be confusing if you tried to add
> "refs/remotes/origin/notes/ with the already existing refs/remotes/origin/notes branch that existed.
>
> Instead, lets add support for a new refs/tracking/* hierarchy which is
> laid out in such a way to avoid this inconsistency. All refs in
> "refs/tracking//*" will include the complete ref, such that
> dropping the "tracking/" part will give the exact ref name as it
> is found in the upstream. Thus, we can track any ref here by simply
> fetching it into refs/tracking//*.
>
> Add support to tell a new remote to start tracking remote hierarchies
> via "--follow" which will track all refs under that section, ie:
>
> git remote add --follow notes origin  will cause
>
> +refs/notes/*:refs/tracking/origin/notes/* to be added as a fetch
> refspec for this remote.
>
> This ensures that any future refs which want a method of sharing the
> current remote status separate from local status could use
> refs/tracking
>
> A long term goal might be to deprecate refs/remotes/ in favor of
> refs/tracking (possibly adding in magic to convert refs/remotes directly
> into refs/tracking so that old stuff still works?).
>
> Things not yet thought through:
>
> 1) maybe we should create a configurable default so that some known set
>of default refs get pulled (ie: notes, grafts, replace, etc?)
>Possibly with some sort of easy way to add or subtract from the list
>in config...
>
> 2) so far, there's no work done to figure out how to remove
>refs/tracking when a remote is dropped. I *think* we can just delete
>all refs under refs/tracking/ but I'm not completely certain
>
> 3) adding magic to complete refs under tracking, such as for git-notes
>or similar may wish to understand shorthand for referencing the
>remote version of notes
>
> 4) adding support for clone.. (this is weird because git-clone and
>git-remote don't both use the same flow for setup.. oops??)
>
> 5) tests, documentation etc. (This is primarily an RFC, with the goal of
>providing a known location for remote references such as notes to
>reside)
>
> 6) we probably want to enable notes and grafts and other similar things
>to use the remote tracking data if its available.
>
> 7) what about tags? Currently we fetch all tags into refs/tags directly,
>which is a bit awkward, if for example you create a local tag and
>a remote creates a tag with the same name, you simply don't get that
>new version. This is good, but now you have no idea how to tell if
>your tags are out of date or not. refs/tracking can partially resolve
>this since remote tags will always be "exactly" what the remote has.
>Unfortunately, I don't know how we'd resolve them into local tags...
>

Oops:

8) if we decided to go with "all refs always get tracked in
refs/tracking" we probably want to either add a way to say "all refs
except refs/tracking ones" or we want a way for servers to (by
default) not advertise refs/tracking when clients fetch from them.
That's partially why I went with the easier "only some hierarchies
will get pulled by default" Otherwise, two remotes that fetch from
each other could create a never ending cycle of
refs/tracking/origin/tracking/origin/tracking/origin/ adding a new
layer every time they fetched.

Thanks,
Jake

[RFC PATCH] proposal for refs/tracking/* hierarchy

[Jacob Keller]

From: Jacob Keller 

Historically, git has tracked the status of remote branches (heads) in
refs/remotes//*. This is necessary and useful as it allows users
to track the difference between their local work and the last known
status of the remote work.

Unfortunately this hierarchy is limited in that it only tracks branches.
Additionally, it is not feasible to extend it directly, because the
layout would become ambiguous. For example, if a user happened to have
a local branch named "notes" it could be confusing if you tried to add
"refs/remotes/origin/notes//*" will include the complete ref, such that
dropping the "tracking/" part will give the exact ref name as it
is found in the upstream. Thus, we can track any ref here by simply
fetching it into refs/tracking//*.

Add support to tell a new remote to start tracking remote hierarchies
via "--follow" which will track all refs under that section, ie:

git remote add --follow notes origin  will cause

+refs/notes/*:refs/tracking/origin/notes/* to be added as a fetch
refspec for this remote.

This ensures that any future refs which want a method of sharing the
current remote status separate from local status could use
refs/tracking

A long term goal might be to deprecate refs/remotes/ in favor of
refs/tracking (possibly adding in magic to convert refs/remotes directly
into refs/tracking so that old stuff still works?).

Things not yet thought through:

1) maybe we should create a configurable default so that some known set
   of default refs get pulled (ie: notes, grafts, replace, etc?)
   Possibly with some sort of easy way to add or subtract from the list
   in config...

2) so far, there's no work done to figure out how to remove
   refs/tracking when a remote is dropped. I *think* we can just delete
   all refs under refs/tracking/ but I'm not completely certain

3) adding magic to complete refs under tracking, such as for git-notes
   or similar may wish to understand shorthand for referencing the
   remote version of notes

4) adding support for clone.. (this is weird because git-clone and
   git-remote don't both use the same flow for setup.. oops??)

5) tests, documentation etc. (This is primarily an RFC, with the goal of
   providing a known location for remote references such as notes to
   reside)

6) we probably want to enable notes and grafts and other similar things
   to use the remote tracking data if its available.

7) what about tags? Currently we fetch all tags into refs/tags directly,
   which is a bit awkward, if for example you create a local tag and
   a remote creates a tag with the same name, you simply don't get that
   new version. This is good, but now you have no idea how to tell if
   your tags are out of date or not. refs/tracking can partially resolve
   this since remote tags will always be "exactly" what the remote has.
   Unfortunately, I don't know how we'd resolve them into local tags...

Probably other things missing too...

Signed-off-by: Jacob Keller 
---
 builtin/remote.c | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/builtin/remote.c b/builtin/remote.c
index f1a88fe26589..dffe3892be11 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -122,6 +122,16 @@ static void add_branch(const char *key, const char 
*branchname,
git_config_set_multivar(key, tmp->buf, "^$", 0);
 }
 
+static void add_tracking(const char *key, const char *namespace,
+const char *remotename, struct strbuf *tmp)
+{
+   strbuf_reset(tmp);
+   strbuf_addch(tmp, '+');
+   strbuf_addf(tmp, "refs/%s/*:refs/tracking/%s/%s/*",
+   namespace, remotename, namespace);
+   git_config_set_multivar(key, tmp->buf, "^$", 0);
+}
+
 static const char mirror_advice[] =
 N_("--mirror is dangerous and deprecated; please\n"
"\t use --mirror=fetch or --mirror=push instead");
@@ -149,6 +159,7 @@ static int add(int argc, const char **argv)
int fetch = 0, fetch_tags = TAGS_DEFAULT;
unsigned mirror = MIRROR_NONE;
struct string_list track = STRING_LIST_INIT_NODUP;
+   struct string_list follow = STRING_LIST_INIT_NODUP;
const char *master = NULL;
struct remote *remote;
struct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;
@@ -164,6 +175,8 @@ static int add(int argc, const char **argv)
N_("or do not fetch any tag at all (--no-tags)"), 
TAGS_UNSET),
OPT_STRING_LIST('t', "track", , N_("branch"),
N_("branch(es) to track")),
+   OPT_STRING_LIST(0, "follow", , N_("namespace"),
+   N_("refs namespaces to follow in 
refs/tracking")),
OPT_STRING('m', "master", , N_("branch"), N_("master 
branch")),
{ OPTION_CALLBACK, 0, "mirror", , N_("push|fetch"),
N_("set up remote as a mirror to push to or fetch 
from"),
@@ -207,6 +220,11 

Re: proposal for how to share other refs as part of refs/tracking/*

[Jacob Keller]

On Tue, Jun 13, 2017 at 8:55 AM, Marc Branchaud <marcn...@xiplink.com> wrote:
> On 2017-06-13 10:41 AM, Marc Branchaud wrote:
>>
>>
>> So I like your refs/tracking proposal, and hope that it aims for mirroring
>> a remote's refs, to eventually replace refs/remotes entirely.
>
>
> To be extra-clear:
>
> I think a refs/tracking hierarchy that starts with notes and maybe some
> other bits is a good first step.
>
> I *hope* such a first step can eventually mirror all of a remote's refs,
> including heads and tags.
>
> I in no way think that this effort should begin by tackling heads and tags.
>
> M.
>

As far as I understand it, mirroring all refs should be as simple as
adding something like

refs/*:refs/tracking//*

assuming that * begins working for multiple components..

Now actually making *use* of that feature would be different.

However, this was generally the direction I was thinking of heading.

Thanks,
Jake

Re: proposal for how to share other refs as part of refs/tracking/*

[Marc Branchaud]

On 2017-06-13 10:41 AM, Marc Branchaud wrote:


So I like your refs/tracking proposal, and hope that it aims for 
mirroring a remote's refs, to eventually replace refs/remotes entirely.


To be extra-clear:

I think a refs/tracking hierarchy that starts with notes and maybe some 
other bits is a good first step.


I *hope* such a first step can eventually mirror all of a remote's refs, 
including heads and tags.


I in no way think that this effort should begin by tackling heads and tags.

M.

Re: proposal for how to share other refs as part of refs/tracking/*

[Marc Branchaud]

On 2017-06-12 06:58 PM, Jacob Keller wrote:

Hi,

There's no actual code yet, (forgive me), but I've been thinking back
to a while ago about attempting to find a way to share things like
refs/notes, and similar refs which are usually not shared across a
remote.

By default, those refs are not propagated when you do a push or a
pull, and this makes using them in any project which has more then one
repository quite difficult.

I'm going to focus the discussion primarily on refs/notes as this is
what I am most interested in, but I think similar issues exist for
refs/grafts and refs/replace, and maybe other sources?


More formal support for custom ref namespaces would be a boon.  For 
example, we have our nightly builds create a "build/w.x.y-z" ref (we 
only want to tag official releases).  Sharing those refs is not hard, 
but a bit obscure.



For branches, we already have a system to share the status of remote
branches, called "refs/remotes//"

This hierarchy unfortunately does not keep space for non-branches,
because you can't simply add a "refs/remotes/notes/<>" or
"refs/remotes//notes" to this as it would be ambiguous.

Now, you might just decide to push the refs/notes directly, ie:

git push origin refs/notes/...:refs/notes/...

Unfortunately, this has problems because it leaves no standard way to
distinguish your local work from what is on the remote, so you can't
easily merge the remote work into yours.


There was a related discussion in the run-up to 1.8.0, about a "remote 
tag namespace" to support having different remotes with the same tag 
name for different objects.  See these messages and their surrounding 
threads:


http://public-inbox.org/git/AANLkTikeqsg+qJ0z4iQ6ZmKL=_HB8YX_z20L=dffa...@mail.gmail.com/

http://public-inbox.org/git/AANLkTi=yfwoaqmhhvlsb1_xmyoe9hhp2yb4h4tqzw...@mail.gmail.com/

http://public-inbox.org/git/201102020322.00171.jo...@herland.net/

The discussion explored, among other things, making 
refs/remotes/$remote/* a mirror of a remote's own refs/* hierarchy 
(well, within reason -- I think there are limits to what should be 
mirrored).


So I like your refs/tracking proposal, and hope that it aims for 
mirroring a remote's refs, to eventually replace refs/remotes entirely.


M.



For example, if Alice creates a new note and pushes it, then Bob
creates a different note on the same commit, he needs to be able to
merge Alice's changes into his note, just like one would do when two
people commit to the same branch.

Today, he must pull the remote notes into a separate location, (since
pulling directly into refs/notes will overwrite his work), and then
update, and then push.

Because this is not standardized, it becomes unwieldy to actually
perform on a day to day basis.

I propose that we add a new "refs/tracking" hierarchy which will be
used to track these type of refs

We could even (long term) migrate refs/remotes into refs/tracking
instead, or provide both with the refs/remotes being pointers or
something like that..

Essentially, refs/notes would be pulled into
refs/tracking//notes/* or something along these lines.

Then, git notes would be modified to be able to have commands to
"pull" and "push" notes which would fetch the remote, and then attempt
a merge into the local notes, while a push would update the remote.

I chose "tracking" because it sort of fits the concept and does not
include things like "remote-notes" or "remotes-v2" which are a bit
weird.

I'm posting this on the mailing list prior to having code because I
wanted to get a sense of how people felt about the question and
whether others still felt it was an issue.

Essentially the goal is to standardize how any refs namespace can be
shared in such a way that local copies can tell what the remote had at
a fetch time, in order to allow easier handling of conflicts between
local and remote changes, just like we do for branches (heads) but
generalized so that other refs namespaces can get the same ability to
handle conflicts.

Thanks,
Jake

proposal for how to share other refs as part of refs/tracking/*

[Jacob Keller]

Hi,

There's no actual code yet, (forgive me), but I've been thinking back
to a while ago about attempting to find a way to share things like
refs/notes, and similar refs which are usually not shared across a
remote.

By default, those refs are not propagated when you do a push or a
pull, and this makes using them in any project which has more then one
repository quite difficult.

I'm going to focus the discussion primarily on refs/notes as this is
what I am most interested in, but I think similar issues exist for
refs/grafts and refs/replace, and maybe other sources?

For branches, we already have a system to share the status of remote
branches, called "refs/remotes//"

This hierarchy unfortunately does not keep space for non-branches,
because you can't simply add a "refs/remotes/notes/<>" or
"refs/remotes//notes" to this as it would be ambiguous.

Now, you might just decide to push the refs/notes directly, ie:

git push origin refs/notes/...:refs/notes/...

Unfortunately, this has problems because it leaves no standard way to
distinguish your local work from what is on the remote, so you can't
easily merge the remote work into yours.

For example, if Alice creates a new note and pushes it, then Bob
creates a different note on the same commit, he needs to be able to
merge Alice's changes into his note, just like one would do when two
people commit to the same branch.

Today, he must pull the remote notes into a separate location, (since
pulling directly into refs/notes will overwrite his work), and then
update, and then push.

Because this is not standardized, it becomes unwieldy to actually
perform on a day to day basis.

I propose that we add a new "refs/tracking" hierarchy which will be
used to track these type of refs

We could even (long term) migrate refs/remotes into refs/tracking
instead, or provide both with the refs/remotes being pointers or
something like that..

Essentially, refs/notes would be pulled into
refs/tracking//notes/* or something along these lines.

Then, git notes would be modified to be able to have commands to
"pull" and "push" notes which would fetch the remote, and then attempt
a merge into the local notes, while a push would update the remote.

I chose "tracking" because it sort of fits the concept and does not
include things like "remote-notes" or "remotes-v2" which are a bit
weird.

I'm posting this on the mailing list prior to having code because I
wanted to get a sense of how people felt about the question and
whether others still felt it was an issue.

Essentially the goal is to standardize how any refs namespace can be
shared in such a way that local copies can tell what the remote had at
a fetch time, in order to allow easier handling of conflicts between
local and remote changes, just like we do for branches (heads) but
generalized so that other refs namespaces can get the same ability to
handle conflicts.

Thanks,
Jake

Good day I write to you in hope that you will be open minded to consider a business partnership proposal in transaction deal worth millions. All I require from you is honesty and an assurance that you

[Ben Moore]

Re: Proposal for missing blob support in Git repos

[Jonathan Tan]

On 05/03/2017 09:29 PM, Junio C Hamano wrote:

Jonathan Tan <jonathanta...@google.com> writes:


I see the semantics as "don't write what you already have", where
"have" means what you have in local storage, but if you extend "have"
to what upstream has, then yes, you're right that this changes
(ignoring shallow clones).

This does remove a resistance that we have against hash collision (in
that normally we would have the correct object for a given hash and
can resist other servers trying to introduce a wrong object, but now
that is no longer the case), but I think it's better than consulting
the hook whenever you want to write anything (which is also a change
in semantics in that you're consulting an external source whenever
you're writing an object, besides the performance implications).


As long as the above pros-and-cons analysis is understood and we are
striking a balance between performance and strictness with such an
understanding of the implications, I am perfectly fine with the
proposal.  That is why my comment has never been "I think that is
wrong" but consistently was "I wonder if that is a good thing."

Thanks.


Noted - if/when I update the patch, I'll include this information in.

Re: Proposal for missing blob support in Git repos

[Junio C Hamano]

Jonathan Tan <jonathanta...@google.com> writes:

> I see the semantics as "don't write what you already have", where
> "have" means what you have in local storage, but if you extend "have"
> to what upstream has, then yes, you're right that this changes
> (ignoring shallow clones).
>
> This does remove a resistance that we have against hash collision (in
> that normally we would have the correct object for a given hash and
> can resist other servers trying to introduce a wrong object, but now
> that is no longer the case), but I think it's better than consulting
> the hook whenever you want to write anything (which is also a change
> in semantics in that you're consulting an external source whenever
> you're writing an object, besides the performance implications).

As long as the above pros-and-cons analysis is understood and we are
striking a balance between performance and strictness with such an
understanding of the implications, I am perfectly fine with the
proposal.  That is why my comment has never been "I think that is
wrong" but consistently was "I wonder if that is a good thing."

Thanks.

Re: Proposal for missing blob support in Git repos

[Jonathan Tan]

On 05/02/2017 11:32 AM, Ævar Arnfjörð Bjarmason wrote:

On Tue, May 2, 2017 at 7:21 PM, Jonathan Tan  wrote:

On Mon, May 1, 2017 at 6:41 PM, Junio C Hamano  wrote:

Jonathan Tan  writes:


On 05/01/2017 04:29 PM, Junio C Hamano wrote:

Jonathan Tan  writes:


Thanks for your comments. If you're referring to the codepath
involving write_sha1_file() (for example, builtin/hash-object ->
index_fd or builtin/unpack-objects), that is fine because
write_sha1_file() invokes freshen_packed_object() and
freshen_loose_object() directly to check if the object already exists
(and thus does not invoke the new mechanism in this patch).


Is that a good thing, though?  It means that you an attacker can
feed one version to the remote object store your "grab blob" hook
gets the blobs from, and have you add a colliding object locally,
and the usual "are we recording the same object as existing one?"
check is bypassed.


If I understand this correctly, what you mean is the situation where
the hook adds an object to the local repo, overriding another object
of the same name?


No.

write_sha1_file() pays attention to objects already in the local
object store to avoid hash collisions that can be used to replace a
known-to-be-good object and that is done as a security measure.
What I am reading in your response was that this new mechanism
bypasses that, and I was wondering if that is a good thing.


Oh, what I meant was that write_sha1_file() bypasses the new
mechanism, not that the new mechanism bypasses the checks in
write_sha1_file().

To be clear, here's what happens when write_sha1_file() is invoked
(before and after this patch - this patch does not affect
write_sha1_file at all):
1. (some details omitted)
2. call freshen_packed_object
3, call freshen_loose_object if necessary
4. write object (if freshen_packed_object and freshen_loose_object do
not both return 0)

Nothing changes in this patch (whether a hook is defined or not).


But don't the semantics change in the sense that before
core.missingBlobCommand you couldn't write a new blob SHA1 that was
already part of your history,


Strictly speaking, you can already do this if you don't have the blob in 
your local repo (for example, with shallow clones - you likely wouldn't 
have blobs pointed to by historical commits outside whatever depth is set).


> whereas with this change

write_sha1_file() might write what it considers to be a new blob, but
it's actually colliding with an existing blob, but write_sha1_file()
doesn't know that because knowing would involve asking the hook to
fetch the blob?


Yes, this might happen.

I see the semantics as "don't write what you already have", where "have" 
means what you have in local storage, but if you extend "have" to what 
upstream has, then yes, you're right that this changes (ignoring shallow 
clones).


This does remove a resistance that we have against hash collision (in 
that normally we would have the correct object for a given hash and can 
resist other servers trying to introduce a wrong object, but now that is 
no longer the case), but I think it's better than consulting the hook 
whenever you want to write anything (which is also a change in semantics 
in that you're consulting an external source whenever you're writing an 
object, besides the performance implications).



And here's what happens when has_sha1_file (or another function listed
in the commit message) is invoked:
1. check for existence of packed object of the requested name
2. check for existence of loose object of the requested name
3. check again for existence of packed object of the requested name
4. if a hook is defined, invoke the hook and repeat 1-3

Here, in step 4, the hook could do whatever it wants to the repository.


This might be a bit of early bikeshedding, but then again the lack of
early bikeshedding tends to turn into standards.

Wouldn't it be better to name this core.missingObjectCommand & have
the hook take a list on stdin like:

[]

And have the hook respond:

[]

I.e. what you'd do now is send this to the hook:

1  blobmissing

And the hook would respond:

1  ok

But this leaves open the door addressing this potential edge case with
writing new blobs in the future, i.e. write_sha1_file() could call it
as:

1  blobnew

And the hook could either respond immediately as:

1  ok

If it's in some #YOLO mode where it's not going to check for colliding
blobs over the network, or alternatively & ask the parent repo if it
has those blobs, and if so print:

1  collision

Or something like that.

This also enables future lazy loading of trees/commits from the same
API, and for the hook to respond out-of-order to the input it gets as
it can, since each request is prefixed with an incrementing request
id.


My initial thought is that it would be better to extend hook support by 
adding 

Re: Proposal for missing blob support in Git repos

[Ævar Arnfjörð Bjarmason]

On Tue, May 2, 2017 at 7:21 PM, Jonathan Tan  wrote:
> On Mon, May 1, 2017 at 6:41 PM, Junio C Hamano  wrote:
>> Jonathan Tan  writes:
>>
>>> On 05/01/2017 04:29 PM, Junio C Hamano wrote:
 Jonathan Tan  writes:

> Thanks for your comments. If you're referring to the codepath
> involving write_sha1_file() (for example, builtin/hash-object ->
> index_fd or builtin/unpack-objects), that is fine because
> write_sha1_file() invokes freshen_packed_object() and
> freshen_loose_object() directly to check if the object already exists
> (and thus does not invoke the new mechanism in this patch).

 Is that a good thing, though?  It means that you an attacker can
 feed one version to the remote object store your "grab blob" hook
 gets the blobs from, and have you add a colliding object locally,
 and the usual "are we recording the same object as existing one?"
 check is bypassed.
>>>
>>> If I understand this correctly, what you mean is the situation where
>>> the hook adds an object to the local repo, overriding another object
>>> of the same name?
>>
>> No.
>>
>> write_sha1_file() pays attention to objects already in the local
>> object store to avoid hash collisions that can be used to replace a
>> known-to-be-good object and that is done as a security measure.
>> What I am reading in your response was that this new mechanism
>> bypasses that, and I was wondering if that is a good thing.
>
> Oh, what I meant was that write_sha1_file() bypasses the new
> mechanism, not that the new mechanism bypasses the checks in
> write_sha1_file().
>
> To be clear, here's what happens when write_sha1_file() is invoked
> (before and after this patch - this patch does not affect
> write_sha1_file at all):
> 1. (some details omitted)
> 2. call freshen_packed_object
> 3, call freshen_loose_object if necessary
> 4. write object (if freshen_packed_object and freshen_loose_object do
> not both return 0)
>
> Nothing changes in this patch (whether a hook is defined or not).

But don't the semantics change in the sense that before
core.missingBlobCommand you couldn't write a new blob SHA1 that was
already part of your history, whereas with this change
write_sha1_file() might write what it considers to be a new blob, but
it's actually colliding with an existing blob, but write_sha1_file()
doesn't know that because knowing would involve asking the hook to
fetch the blob?

> And here's what happens when has_sha1_file (or another function listed
> in the commit message) is invoked:
> 1. check for existence of packed object of the requested name
> 2. check for existence of loose object of the requested name
> 3. check again for existence of packed object of the requested name
> 4. if a hook is defined, invoke the hook and repeat 1-3
>
> Here, in step 4, the hook could do whatever it wants to the repository.

This might be a bit of early bikeshedding, but then again the lack of
early bikeshedding tends to turn into standards.

Wouldn't it be better to name this core.missingObjectCommand & have
the hook take a list on stdin like:

[]

And have the hook respond:

[]

I.e. what you'd do now is send this to the hook:

1  blobmissing

And the hook would respond:

1  ok

But this leaves open the door addressing this potential edge case with
writing new blobs in the future, i.e. write_sha1_file() could call it
as:

1  blobnew

And the hook could either respond immediately as:

1  ok

If it's in some #YOLO mode where it's not going to check for colliding
blobs over the network, or alternatively & ask the parent repo if it
has those blobs, and if so print:

1  collision

Or something like that.

This also enables future lazy loading of trees/commits from the same
API, and for the hook to respond out-of-order to the input it gets as
it can, since each request is prefixed with an incrementing request
id.

Re: Proposal for missing blob support in Git repos

[Junio C Hamano]

Jonathan Tan  writes:

> On 05/01/2017 04:29 PM, Junio C Hamano wrote:
>> Jonathan Tan  writes:
>>
>>> Thanks for your comments. If you're referring to the codepath
>>> involving write_sha1_file() (for example, builtin/hash-object ->
>>> index_fd or builtin/unpack-objects), that is fine because
>>> write_sha1_file() invokes freshen_packed_object() and
>>> freshen_loose_object() directly to check if the object already exists
>>> (and thus does not invoke the new mechanism in this patch).
>>
>> Is that a good thing, though?  It means that you an attacker can
>> feed one version to the remote object store your "grab blob" hook
>> gets the blobs from, and have you add a colliding object locally,
>> and the usual "are we recording the same object as existing one?"
>> check is bypassed.
>
> If I understand this correctly, what you mean is the situation where
> the hook adds an object to the local repo, overriding another object
> of the same name?

No.  

write_sha1_file() pays attention to objects already in the local
object store to avoid hash collisions that can be used to replace a
known-to-be-good object and that is done as a security measure.
What I am reading in your response was that this new mechanism
bypasses that, and I was wondering if that is a good thing.

Re: Proposal for missing blob support in Git repos

[Brandon Williams]

On 05/01, Jonathan Tan wrote:
> On 05/01/2017 04:29 PM, Junio C Hamano wrote:
> >Jonathan Tan  writes:
> >
> >>Thanks for your comments. If you're referring to the codepath
> >>involving write_sha1_file() (for example, builtin/hash-object ->
> >>index_fd or builtin/unpack-objects), that is fine because
> >>write_sha1_file() invokes freshen_packed_object() and
> >>freshen_loose_object() directly to check if the object already exists
> >>(and thus does not invoke the new mechanism in this patch).
> >
> >Is that a good thing, though?  It means that you an attacker can
> >feed one version to the remote object store your "grab blob" hook
> >gets the blobs from, and have you add a colliding object locally,
> >and the usual "are we recording the same object as existing one?"
> >check is bypassed.
> 
> If I understand this correctly, what you mean is the situation where
> the hook adds an object to the local repo, overriding another object
> of the same name? If yes, I think that is the nature of executing an
> arbitrary command. If we really want to avoid that, we could drop
> the hook functionality (and instead, for example, provide the URL of
> a Git repo instead from which we can communicate using a new
> fetch-blob protocol), although that would reduce the usefulness of
> this, especially during the transition period in which we don't have
> any sort of batching of requests.

If I understand correctly this is where we aim to be once all is said
and done.  I guess the question is what are we willing to do during the
transition phase.

-- 
Brandon Williams

Re: Proposal for missing blob support in Git repos

[Jonathan Tan]

On 05/01/2017 04:29 PM, Junio C Hamano wrote:

Jonathan Tan  writes:


Thanks for your comments. If you're referring to the codepath
involving write_sha1_file() (for example, builtin/hash-object ->
index_fd or builtin/unpack-objects), that is fine because
write_sha1_file() invokes freshen_packed_object() and
freshen_loose_object() directly to check if the object already exists
(and thus does not invoke the new mechanism in this patch).


Is that a good thing, though?  It means that you an attacker can
feed one version to the remote object store your "grab blob" hook
gets the blobs from, and have you add a colliding object locally,
and the usual "are we recording the same object as existing one?"
check is bypassed.


If I understand this correctly, what you mean is the situation where the 
hook adds an object to the local repo, overriding another object of the 
same name? If yes, I think that is the nature of executing an arbitrary 
command. If we really want to avoid that, we could drop the hook 
functionality (and instead, for example, provide the URL of a Git repo 
instead from which we can communicate using a new fetch-blob protocol), 
although that would reduce the usefulness of this, especially during the 
transition period in which we don't have any sort of batching of requests.

Re: Proposal for missing blob support in Git repos

[Junio C Hamano]

Jonathan Tan  writes:

> Thanks for your comments. If you're referring to the codepath
> involving write_sha1_file() (for example, builtin/hash-object ->
> index_fd or builtin/unpack-objects), that is fine because
> write_sha1_file() invokes freshen_packed_object() and
> freshen_loose_object() directly to check if the object already exists
> (and thus does not invoke the new mechanism in this patch).

Is that a good thing, though?  It means that you an attacker can
feed one version to the remote object store your "grab blob" hook
gets the blobs from, and have you add a colliding object locally,
and the usual "are we recording the same object as existing one?"
check is bypassed.

Re: Proposal for missing blob support in Git repos

[Jonathan Tan]

On 04/30/2017 08:57 PM, Junio C Hamano wrote:

One thing I wonder is what the performance impact of a change like
this to the codepath that wants to see if an object does _not_ exist
in the repository.  When creating a new object by hashing raw data,
we see if an object with the same name already exists before writing
the compressed loose object out (or comparing the payload to detect
hash collision).  With a "missing blob" support, we'd essentially
spawn an extra process every time we want to create a new blob
locally, and most of the time that is done only to hear the external
command to say "no, we've never heard of such an object", with a
possibly large latency.

If we do not have to worry about that (or if it is no use to worry
about it, because we cannot avoid it if we wanted to do the lazy
loading of objects from elsewhere), then the patch presented here
looked like a sensible first step towards the stated goal.

Thanks.


Thanks for your comments. If you're referring to the codepath involving 
write_sha1_file() (for example, builtin/hash-object -> index_fd or 
builtin/unpack-objects), that is fine because write_sha1_file() invokes 
freshen_packed_object() and freshen_loose_object() directly to check if 
the object already exists (and thus does not invoke the new mechanism in 
this patch).


Having said that, looking at other parts of the fetching mechanism, 
there are a few calls to has_sha1_file() and others that might need to 
be checked. (We have already discussed one - the one in rev-list when 
invoked to check connectivity.) I could take a look at that, but was 
hoping for discussion on what I've sent so far (so that I know that I'm 
on the right track, and because it somewhat works, albeit slowly).

Re: Proposal for missing blob support in Git repos

[Junio C Hamano]

Jonathan Tan  writes:

> In order to determine the code changes in sha1_file.c necessary, I
> investigated the following:
>  (1) functions in sha1_file that take in a hash, without the user
>  regarding how the object is stored (loose or packed)
>  (2) functions in sha1_file that operate on packed objects (because I
>  need to check callers that know about the loose/packed distinction
>  and operate on both differently, and ensure that they can handle
>  the concept of objects that are neither loose nor packed)
>
> For (1), I looked through all non-static functions in sha1_file.c that
> take in an unsigned char * parameter. The ones that are relevant, and my
> modifications to them to resolve this problem, are:
>  - sha1_object_info_extended (fixed in this commit)
>  - sha1_object_info (auto-fixed by sha1_object_info_extended)
>  - read_sha1_file_extended (fixed by fixing read_object)
>  - read_object_with_reference (auto-fixed by read_sha1_file_extended)
>  - force_object_loose (only called from builtin/pack-objects.c, which
>already knows that at least one pack contains this object)
>  - has_sha1_file_with_flags (fixed in this commit)
>  - assert_sha1_type (auto-fixed by sha1_object_info)
>
> As described in the list above, several changes have been included in
> this commit to fix the necessary functions.
>
> For (2), I looked through the same functions as in (1) and also
> for_each_packed_object. The ones that are relevant are:
>  - parse_pack_index
>- http - indirectly from http_get_info_packs
>  - find_pack_entry_one
>- this searches a single pack that is provided as an argument; the
>  caller already knows (through other means) that the sought object
>  is in a specific pack
>  - find_sha1_pack
>- fast-import - appears to be an optimization to not store a
>  file if it is already in a pack
>- http-walker - to search through a struct alt_base
>- http-push - to search through remote packs
>  - has_sha1_pack
>- builtin/fsck - fixed in this commit
>- builtin/count-objects - informational purposes only (check if loose
>  object is also packed)
>- builtin/prune-packed - check if object to be pruned is packed (if
>  not, don't prune it)
>- revision - used to exclude packed objects if requested by user
>- diff - just for optimization
>  - for_each_packed_object
>- reachable - only to find recent objects
>- builtin/fsck - fixed in this commit
>- builtin/cat-file - see below
>
> As described in the list above, builtin/fsck has been updated. I have
> left builtin/cat-file alone; this means that cat-file
> --batch-all-objects will only operate on objects physically in the repo.

One thing I wonder is what the performance impact of a change like
this to the codepath that wants to see if an object does _not_ exist
in the repository.  When creating a new object by hashing raw data,
we see if an object with the same name already exists before writing
the compressed loose object out (or comparing the payload to detect
hash collision).  With a "missing blob" support, we'd essentially
spawn an extra process every time we want to create a new blob
locally, and most of the time that is done only to hear the external
command to say "no, we've never heard of such an object", with a
possibly large latency.

If we do not have to worry about that (or if it is no use to worry
about it, because we cannot avoid it if we wanted to do the lazy
loading of objects from elsewhere), then the patch presented here
looked like a sensible first step towards the stated goal.

Thanks.

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Jonathan Tan]

On 04/21/2017 09:41 AM, Kevin David wrote:

Hi Jonathan,

Sorry for the delayed response - other work got in the way, unfortunately!


No worries!


I am envisioning (1a) as described in Jeff Hostetler's e-mail [1] ("a
pre-command or hook to identify needed blobs and pre-fetch them before
allowing the actual command to start"), so a Git command would typically
make a single request that contains all the blobs required, but my
proposal can also handle (1c) ('"fault" them in as necessary in
read_object() while the command is running and without any pre-fetch
(either synchronously or asynchronously and with/without a helper
process)').

Even if we decided to go with single-blob requests and responses, it is
still important to send them as packfiles, so that the server can serve
them directly from its compressed storage without first having to
uncompress them.

[1] 
https://public-inbox.org/git/1488999039-37631-1-git-send-email-...@jeffhostetler.com/


Ah, I missed this. If we think we can build meaningfully-sized requests via 
(1a) and (1b),
then I agree - packs are optimal.

However, if single-blob requests/responses dominate, I have a few concerns, 
mostly from
experience playing with something similar:
* Regardless of the size of the object or number returned, the client will need 
to
  `index-pack` the result and create a corresponding `.idx` file, which requires
  decompression to construct (right?)


Yes, although I think that loose objects would need to be decompressed 
too to verify the SHA-1.



* Unless the client's repository is repacked aggressively, we'll pollute the
  `.git\objects\pack` directory with little indexes (somewhere around 4KiB 
minimum)
  and packfiles rapidly, which would degrade performance. One rudimentary 
workaround
  would be to loosen these packs on the client if they were under a certain
  size/object count. I think fetch does this already?

In either case, shifting the pack decompression/loose object recompression 
problem
to the client instead of the server is probably a good principle, but in our 
case it
simply wasn't fast enough to serve single blobs interactively (e.g. opening a 
source
file you don't have locally). I'm hopeful that the proposed partial clone 
solutions you
referenced would reduce the frequency of this being required.


I wrote up a proposal for how Git should handle missing blobs, including 
a hook that, when invoked, is allowed to write packed or loose objects 
[1]. I was thinking along similar lines too about the many-packfile 
issue, and that was part of the reason why [1] was written the way it is.


I think that, by default, the wire protocol should transmit packs 
because it causes less load on the server (as you describe) and it 
supports batch requests. Also, this protocol with single-blob fetching 
might be usable as-is for projects that only exclude large blobs (so 
there are relatively few of them), and will be more and more usable as 
we migrate the more commonly used Git commands to provide batch 
requests. However, the ability to customize the hook allows users to use 
another download scheme more suited to their use cases (for example, in 
yours, where you can provision servers that are topologically close to 
your clients).


[1] <20170426221346.25337-1-jonathanta...@google.com>


Being a bit more clever about packing objects (e.g. splitting blobs out from 
commits
and trees) improved this a bit, but we still hit a bottlenecks from what 
appeared to
be a large number of memory-mapping operations on a ~140GiB packfile of blobs.

Each `pack-objects` process would consume approximately one CPU core for the
duration of the request. It's possible that further splitting of these large 
blob packs
would have improved performance in some scenarios, but that would increase the
amount of pack-index lookups necessary to find a single object.


I'm not very experienced with mmap, but I thought that memory-mapping a
large file in itself does not incur much of a performance penalty (if
any) - it is the accesses that count. I experimented with 15,000 and
150,000 MiB files and mmap and they seem to be handled quite well. Also,
how many objects are "pack-objects" packing here?


Back when we took this approach, it was ~4000 blob objects at a time.
Perhaps we were being bitten by the Windows implementation of git_mmap[3]?.
When I profiled the 4k blob scenario, the majority of CPU and wall time was
spent in MapViewOfFileEx, which looks like it could mean accesses as well.

[3] https://github.com/git/git/blob/master/compat/win32mmap.c


Hmm...I don't have much experience with this, but this may be something 
to keep in mind.



This seems like a clever way to avoid the canonical 
`/info/refs?service=git-upload-pack`
capability negotiation on every call. However, using error handling to fallback 
seems
slightly wonky to me. Hopefully users are incentivized to upgrade their clients.


By "error handling

Proposal for missing blob support in Git repos

[Jonathan Tan]

Here is a proposal for missing blob support in Git repos. There have
been several other proposals [1] [2]; this is similar to those except
that I have provided a more comprehensive analysis of the changes that
need to be done, and have made those changes (see commit below the
scissors line).

This proposal is limited to local handling using a user-specified hook.
I have another proposal out [3] for what a potential default hook should
be - how Git on the server can serve blobs to repos that have missing
blobs.

[1] <20170304191901.9622-1-mar...@efaref.net>
[2] <1488999039-37631-1-git-send-email-...@jeffhostetler.com>
[3] <ffd92ad9-39fe-c76b-178d-6e3d6a425...@google.com>

-- 8< --
sha1_file, fsck: add missing blob hook support

Currently, Git does not support repos with very large numbers of blobs
or repos that wish to minimize manipulation of certain blobs (for
example, because they are very large) very well, even if the user
operates mostly on part of the repo, because Git is designed on the
assumption that every blob referenced by a tree object is available
somewhere in the repo storage.

As a first step to reducing this problem, add rudimentary support for
missing blobs by teaching sha1_file to invoke a hook whenever a blob is
requested but missing, and by updating fsck to tolerate missing blobs.
The hook is a shell command that can be configured through "git config";
this hook takes in a list of hashes and writes (if successful) the
corresponding objects to the repo's local storage.

This commit does not include support for generating such a repo; neither
has any command (other than fsck) been modified to either tolerate
missing blobs (without invoking the hook) or be more efficient in
invoking the missing blob hook. Only a fallback is provided in the form
of sha1_file invoking the missing blob hook when necessary.

In order to determine the code changes in sha1_file.c necessary, I
investigated the following:
 (1) functions in sha1_file that take in a hash, without the user
 regarding how the object is stored (loose or packed)
 (2) functions in sha1_file that operate on packed objects (because I
 need to check callers that know about the loose/packed distinction
 and operate on both differently, and ensure that they can handle
 the concept of objects that are neither loose nor packed)

For (1), I looked through all non-static functions in sha1_file.c that
take in an unsigned char * parameter. The ones that are relevant, and my
modifications to them to resolve this problem, are:
 - sha1_object_info_extended (fixed in this commit)
 - sha1_object_info (auto-fixed by sha1_object_info_extended)
 - read_sha1_file_extended (fixed by fixing read_object)
 - read_object_with_reference (auto-fixed by read_sha1_file_extended)
 - force_object_loose (only called from builtin/pack-objects.c, which
   already knows that at least one pack contains this object)
 - has_sha1_file_with_flags (fixed in this commit)
 - assert_sha1_type (auto-fixed by sha1_object_info)

As described in the list above, several changes have been included in
this commit to fix the necessary functions.

For (2), I looked through the same functions as in (1) and also
for_each_packed_object. The ones that are relevant are:
 - parse_pack_index
   - http - indirectly from http_get_info_packs
 - find_pack_entry_one
   - this searches a single pack that is provided as an argument; the
 caller already knows (through other means) that the sought object
 is in a specific pack
 - find_sha1_pack
   - fast-import - appears to be an optimization to not store a
 file if it is already in a pack
   - http-walker - to search through a struct alt_base
   - http-push - to search through remote packs
 - has_sha1_pack
   - builtin/fsck - fixed in this commit
   - builtin/count-objects - informational purposes only (check if loose
 object is also packed)
   - builtin/prune-packed - check if object to be pruned is packed (if
 not, don't prune it)
   - revision - used to exclude packed objects if requested by user
   - diff - just for optimization
 - for_each_packed_object
   - reachable - only to find recent objects
   - builtin/fsck - fixed in this commit
   - builtin/cat-file - see below

As described in the list above, builtin/fsck has been updated. I have
left builtin/cat-file alone; this means that cat-file
--batch-all-objects will only operate on objects physically in the repo.

Some alternative designs that I considered but rejected:

 - Storing a list of hashes of missing blobs, possibly with metadata
   (much like the shallow list). Having such a list would allow for
   things like better error messages, attaching metadata (for example,
   file size or binary/text nature) to each blob, and configuring
   different hooks for each blob, but it is difficult to scale to large
   repos.
 - Adding a hook whenever a packed blob is requested, not on any blob.
   That is, whenever we attempt to 

RE: Proposal for "fetch-any-blob Git protocol" and server design

[Kevin David]

Hi Jonathan,

Sorry for the delayed response - other work got in the way, unfortunately!

Kevin

-Original Message-
From: Jonathan Tan [mailto:jonathanta...@google.com] 
Sent: Thursday, April 13, 2017 4:13 PM

>> I know we're considering server behavior here, but how large do you generally
>> expect these blob-want requests to be? I ask because we took an initial 
>> approach
>> very similar to this, however, we had a hard time being clever about 
>> figuring out
>> what set of blobs to request for those clients that didn't want the entire 
>> set, and
>> ended up falling back to single-blob requests.
>>
>> Obviously, this could be due to thenature of our 
>> filesystem-virtualization-based client,
>> but I also suspect that the teams attacking this problem are more often than 
>> not dealing
>> with very large blob objects, so the cost of a round-trip becomes lower 
>> relative to sending
>> the object content itself.
>
> I am envisioning (1a) as described in Jeff Hostetler's e-mail [1] ("a 
> pre-command or hook to identify needed blobs and pre-fetch them before 
> allowing the actual command to start"), so a Git command would typically 
> make a single request that contains all the blobs required, but my 
> proposal can also handle (1c) ('"fault" them in as necessary in 
> read_object() while the command is running and without any pre-fetch 
> (either synchronously or asynchronously and with/without a helper 
> process)').
> 
> Even if we decided to go with single-blob requests and responses, it is 
> still important to send them as packfiles, so that the server can serve 
> them directly from its compressed storage without first having to 
> uncompress them.
> 
> [1] 
> https://public-inbox.org/git/1488999039-37631-1-git-send-email-...@jeffhostetler.com/

Ah, I missed this. If we think we can build meaningfully-sized requests via 
(1a) and (1b), 
then I agree - packs are optimal.

However, if single-blob requests/responses dominate, I have a few concerns, 
mostly from 
experience playing with something similar: 
* Regardless of the size of the object or number returned, the client will need 
to 
  `index-pack` the result and create a corresponding `.idx` file, which 
requires 
  decompression to construct (right?)

* Unless the client's repository is repacked aggressively, we'll pollute the 
  `.git\objects\pack` directory with little indexes (somewhere around 4KiB 
minimum) 
  and packfiles rapidly, which would degrade performance. One rudimentary 
workaround 
  would be to loosen these packs on the client if they were under a certain 
  size/object count. I think fetch does this already?

In either case, shifting the pack decompression/loose object recompression 
problem 
to the client instead of the server is probably a good principle, but in our 
case it 
simply wasn't fast enough to serve single blobs interactively (e.g. opening a 
source 
file you don't have locally). I'm hopeful that the proposed partial clone 
solutions you 
referenced would reduce the frequency of this being required.

>> Being a bit more clever about packing objects (e.g. splitting blobs out from 
>> commits
>> and trees) improved this a bit, but we still hit a bottlenecks from what 
>> appeared to
>> be a large number of memory-mapping operations on a ~140GiB packfile of 
>> blobs.
>> 
>> Each `pack-objects` process would consume approximately one CPU core for the
>> duration of the request. It's possible that further splitting of these large 
>> blob packs
>> would have improved performance in some scenarios, but that would increase 
>> the
>> amount of pack-index lookups necessary to find a single object.
>
> I'm not very experienced with mmap, but I thought that memory-mapping a 
> large file in itself does not incur much of a performance penalty (if 
> any) - it is the accesses that count. I experimented with 15,000 and 
> 150,000 MiB files and mmap and they seem to be handled quite well. Also, 
> how many objects are "pack-objects" packing here?

Back when we took this approach, it was ~4000 blob objects at a time. 
Perhaps we were being bitten by the Windows implementation of git_mmap[3]?. 
When I profiled the 4k blob scenario, the majority of CPU and wall time was 
spent in MapViewOfFileEx, which looks like it could mean accesses as well.

[3] https://github.com/git/git/blob/master/compat/win32mmap.c

>> This seems like a clever way to avoid the canonical 
>> `/info/refs?service=git-upload-pack`
>> capability negotiation on every call. However, using error handling to 
>> fallback seems
>> slightly wonky to me. Hopefully users are incentivized to upgrade their 
>> clients.
>
> By &quo

Business Proposal

[QUATIF GROUP OF COMPANIES]

Dear Friend,
 
I would like to discuss a very important issue with you. I am writing to find 
out if this is your valid email. Please, let me know if this email is valid
 
Kind regards
Adrien Saif
Attorney to Quatif Group of Companies

Business Proposal

[QUATIF GROUP OF COMPANIES]

Dear Friend,
 
I would like to discuss a very important issue with you. I am writing to find 
out if this is your valid email. Please, let me know if this email is valid
 
Kind regards
Adrien Saif
Attorney to Quatif Group of Companies

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Jonathan Tan]

On 04/12/2017 03:02 PM, Kevin David wrote:

Hi Jonathan,

I work on the network protocols for the GVFS project at Microsoft.
I shared a couple thoughts and questions below.


Thanks for your reply!


I know we're considering server behavior here, but how large do you generally
expect these blob-want requests to be? I ask because we took an initial approach
very similar to this, however, we had a hard time being clever about figuring 
out
what set of blobs to request for those clients that didn't want the entire set, 
and
ended up falling back to single-blob requests.

Obviously, this could be due to thenature of our 
filesystem-virtualization-based client,
but I also suspect that the teams attacking this problem are more often than 
not dealing
with very large blob objects, so the cost of a round-trip becomes lower 
relative to sending
the object content itself.


I am envisioning (1a) as described in Jeff Hostetler's e-mail [1] ("a 
pre-command or hook to identify needed blobs and pre-fetch them before 
allowing the actual command to start"), so a Git command would typically 
make a single request that contains all the blobs required, but my 
proposal can also handle (1c) ('"fault" them in as necessary in 
read_object() while the command is running and without any pre-fetch 
(either synchronously or asynchronously and with/without a helper 
process)').


Even if we decided to go with single-blob requests and responses, it is 
still important to send them as packfiles, so that the server can serve 
them directly from its compressed storage without first having to 
uncompress them.


[1] 
https://public-inbox.org/git/1488999039-37631-1-git-send-email-...@jeffhostetler.com/



Along the same lines as above, this is where we started and it worked well for
low-volume requests. However, when we started ramping up the load,
`pack-objects` operating on a very large packed repository (~150 GiB) became
very computationally expensive, even with `--compression=1 --depth=0 
--window=0`.

Being a bit more clever about packing objects (e.g. splitting blobs out from 
commits
and trees) improved this a bit, but we still hit a bottlenecks from what 
appeared to
be a large number of memory-mapping operations on a ~140GiB packfile of blobs.

Each `pack-objects` process would consume approximately one CPU core for the
duration of the request. It's possible that further splitting of these large 
blob packs
would have improved performance in some scenarios, but that would increase the
amount of pack-index lookups necessary to find a single object.


I'm not very experienced with mmap, but I thought that memory-mapping a 
large file in itself does not incur much of a performance penalty (if 
any) - it is the accesses that count. I experimented with 15,000 and 
150,000 MiB files and mmap and they seem to be handled quite well. Also, 
how many objects are "pack-objects" packing here?



=== Endpoint support for forward compatibility

This "server" endpoint requires that the first line be understood, but
will ignore any other lines starting with words that it does not
understand. This allows new "commands" to be added (distinguished by
their first lines) and existing commands to be "upgraded" with backwards 
compatibility.


This seems like a clever way to avoid the canonical 
`/info/refs?service=git-upload-pack`
capability negotiation on every call. However, using error handling to fallback 
seems
slightly wonky to me. Hopefully users are incentivized to upgrade their clients.


By "error handling to fallback", do you mean in my proposal or in a 
possible future one (assuming my proposal is implemented)? I don't think 
my proposal requires any error handling to fallback (since only new 
clients can clone partially - old clients will just clone totally and 
obliviously), but I acknowledge that this proposal does not mean that 
any future proposal can be done without requiring error handling to 
fallback.



=== Indication to use the proposed endpoint

The client will probably already record that at least one of its
remotes (the one that it successfully performed a "partial clone"
from) supports this new endpoint (if not, it can’t determine whether a
missing blob was caused by repo corruption or by the "partial clone").
This knowledge can be used both to know that the server supports
"fetch-blob-pack" and "fetch-commit-pack" (for the latter, the client
can fall back to "fetch-pack"/"upload-pack" when fetching from other servers).


This makes a lot of sense to me. When we built our caching proxy, we had to be 
careful
when designing how we'd handle clients requesting objects missing from the 
proxy.

For example, a client requests a single blob and the proxy doesn't have it - we 
can't simply
download that object from the "authoritative" remote and stick it in the 
`.git\objects\xx\yyy...`
di

RE: Proposal for "fetch-any-blob Git protocol" and server design

[Kevin David]

Hi Jonathan, 

I work on the network protocols for the GVFS project at Microsoft.
I shared a couple thoughts and questions below.

Thanks,

Kevin

-Original Message-
From: Stefan Beller [mailto:sbel...@google.com]
Sent: Tuesday, March 28, 2017 7:20 PM
To: Jonathan Tan <jonathanta...@google.com>; Ben Peart <peart...@gmail.com>
Cc: git@vger.kernel.org; Mark Thomas <mar...@efaref.net>; Jeff Hostetler 
<g...@jeffhostetler.com>
Subject: Re: Proposal for "fetch-any-blob Git protocol" and server design

+cc Ben Peart, who sent
"[RFC] Add support for downloading blobs on demand" to the list recently.
This proposal here seems like it has the same goal, so maybe your review could 
go a long way here?

Thanks,
Stefan

On Tue, Mar 14, 2017 at 3:57 PM, Jonathan Tan <jonathanta...@google.com> wrote:
> As described in "Background" below, there have been at least 2 patch 
> sets to support "partial clones" and on-demand blob fetches, where the 
> server part that supports on-demand blob fetches was treated at least 
> in outline. Here is a proposal treating that server part in detail.
>
> == Background
>
> The desire for Git to support (i) missing blobs and (ii) fetching them 
> as needed from a remote repository has surfaced on the mailing list a 
> few times, most recently in the form of RFC patch sets [1] [2].
>
> A local repository that supports (i) will be created by a "partial 
> clone", that is, a clone with some special parameters (exact 
> parameters are still being discussed) that does not download all blobs 
> normally downloaded. Such a repository should support (ii), which is what 
> this proposal describes.
>
> == Design
>
> A new endpoint "server" is created. The client will send a message in 
> the following format:
>
> 
> fbp-request = PKT-LINE("fetch-blob-pack")
>   1*want
>   flush-pkt
> want = PKT-LINE("want" SP obj-id)
> 
>
> The client may send one or more SHA-1s for which it wants blobs, then 
> a flush-pkt.

I know we're considering server behavior here, but how large do you generally
expect these blob-want requests to be? I ask because we took an initial approach
very similar to this, however, we had a hard time being clever about figuring 
out
what set of blobs to request for those clients that didn't want the entire set, 
and
ended up falling back to single-blob requests. 

Obviously, this could be due to thenature of our 
filesystem-virtualization-based client, 
but I also suspect that the teams attacking this problem are more often than 
not dealing
with very large blob objects, so the cost of a round-trip becomes lower 
relative to sending
the object content itself.

>
> The server will then reply:
>
> 
> server-reply = flush-pkt | PKT-LINE("ERR" SP message)
> 
>
> If there was no error, the server will then send them in a packfile, 
> formatted like described in "Packfile Data" in pack-protocol.txt with 
> "side-band-64k" enabled.
> Any server that supports "partial clone" will also support this, and 
> the client will automatically assume this. (How a client discovers 
> "partial clone" is not covered by this proposal.)

Along the same lines as above, this is where we started and it worked well for
low-volume requests. However, when we started ramping up the load,
`pack-objects` operating on a very large packed repository (~150 GiB) became
very computationally expensive, even with `--compression=1 --depth=0 
--window=0`.

Being a bit more clever about packing objects (e.g. splitting blobs out from 
commits
and trees) improved this a bit, but we still hit a bottlenecks from what 
appeared to
be a large number of memory-mapping operations on a ~140GiB packfile of blobs.

Each `pack-objects` process would consume approximately one CPU core for the
duration of the request. It's possible that further splitting of these large 
blob packs
would have improved performance in some scenarios, but that would increase the
amount of pack-index lookups necessary to find a single object.

>
> The server will perform reachability checks on requested blobs through 
> the equivalent of "git rev-list --use-bitmap-index" (like "git 
> upload-pack" when using the allowreachablesha1inwant option), unless 
> configured to suppress reachability checks through a config option.
> The server administrator is highly recommended to regularly regenerate 
> the bitmap (or suppress reachability checks).
>
> === Endpoint support for forward compatibility
>
> This "server" endpoint requires that the first line be understood, but 
> will ignore any other lines starting with words that it does not 
> understand. This a

Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Daniel Ferreira (theiostream)]

On Sun, Apr 2, 2017 at 4:43 PM, Johannes Schindelin
 wrote:
> We ask to accomplish a microproject before evaluating the proposals for
> one reason: to have a good understanding how well the students would
> interact with the project if they were accepted. As such, the
> microprojects really are about the flow of the contribution, not to tackle
> the project already.
> Meaning: I would recommend staying with your microproject, in particular
> if it is already in full swing.

Oh, when I mentioned these bugfixes I meant I'd be willing to do those
*plus* my microproject before the coding period begins as a "warm-up"
to the project. I'm certainly staying with the microproject until the
end!

-- Daniel.

Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Johannes Schindelin]

Hi Daniel,

On Fri, 31 Mar 2017, Daniel Ferreira (theiostream) wrote:

> Question: do you suggest any pending bugfix to git-add--interactive or
> to something related that might give some useful knowledge in advance?
> (for the pre-code period). My microproject involves playing with the
> dir_iterator interface, which is a great exercise in code refactoring
> but really does not teach me too much about Git's architecture.

We ask to accomplish a microproject before evaluating the proposals for
one reason: to have a good understanding how well the students would
interact with the project if they were accepted. As such, the
microprojects really are about the flow of the contribution, not to tackle
the project already.

Meaning: I would recommend staying with your microproject, in particular
if it is already in full swing.

Ciao,
Johannes

[RFC] [GSoC] Proposal Draft for GSoC 2017 : Incremental Rewrite of git-submodules

[Prathamesh Chavan]

Hello everyone,

This is the first draft of my project proposal. I decided to change my project
since the project I initially intended to do and also proposed was slightly
a smaller project for 13 weeks. Also when I came across this project,
and also found out that there are left over bits (git submodule $cmd
$pathspec may want to error out when the $pathspec does not match any
submodules) which also can be included in this conversion process, I was
motivated to change my project and hence made my proposal regarding it.

Also I wish to complete the previously proposed task, probably after finishing
GSoC first.

Here is a Google doc for my proposal,
https://docs.google.com/document/d/1krxVLooWl--75Pot3dazhfygR3wCUUWZWzTXtK1L-xU

Below I have also included the text only version of my proposal, so that
all the discussion related to it would also be there on the public-inbox of git.
It would be great to have your suggestion, so that I can improve it futher.

Thanks,
Prathamesh Chavan
---

Incremental Rewrite of git-submodules
01.04.2017

About Me

Name  Prathamesh Chavan
UniversityIndian Institute of Technology, Kharagpur
Major Computer Science and Engineering(Dual Degree)
Email pc44...@gmail.com
IRC   pratham_pc
Blog  pratham-pc.github.io
Contact No.   +91-993-235-8333
Time Zone IST (UTC +5:30)


Background

I am a second-year student from the Department of Computer Science and
Engineering IIT Kharagpur. I’m an active member at Kharagpur open
source society and also of Kharagpur Linux Users Group since my first
year. I got introduced to open source, Linux and git since the
beginning of my first year.
I use to try to complete any task on my laptop without the use of the
cursor and hence eventually got familiar with shell script and in the
same manner, I still find something new every day. I’m also quite
familiar with C language. I always wanted to get involved in the
development process. As Google Summer of Code is a great way to
achieve it, hence I would like to participate.
This is my first attempt to GSoC. Since I use Git on regular basis
and would also continue to use it, I would love to be a part of its
contributors and hence would also be able to contribute to the project
even after GSoC.

The Project

Abstract

To manage the external dependencies, git submodule commands have been
frequently used by the developers in their projects. But as most of the
subcommands of git submodule are still written in shell scripts (All, but
git submodule init), my project intends to convert the subcommands into
C code, thus making them builtins. This will increase Git's portability
and hence the efficiency of working with the git-submodule commands.
The function cmd_init has been ported to its built-in successfully and
similar operation needs to be done to the remaining functions.

Problems encountered while using Shell Scripts

Git submodule subcommands are currently implemented by using shell script
'git-submodule.sh'. There are several reasons why we'll prefer not to
use the shell script:

1. Less Efficient

Since being a part of shell script, the commands of git submodule do not
have access to the low-level internal API of git, which are designed to
carry out the task more efficiently.
Also, since the subcommands in  ‘git-submodule.sh’ do not have direct
access to the native low-level internal API's of git, carrying out even
trivial processes make git be spawned into a separate process with the
help of plumbing functions. Also since everything comes from external
commands even in the shell scripts, everything needs to undergo through
a fork and exec. Whenever fork takes place, the git command requires
another fully functioning POSIX shell. This results in making the git
submodule commands slow and relatively inefficient, and also reduces git's
portability, since it is difficult to port such commands to a non-POSIX
an environment like Windows.

2. Spawning makes the process slower in many cases

Most of the operations in the shell scripts require the scripts to call
the external executables, even for performing trivial tasks. When we
take a look at the file 'git-submodule.sh', then its is seen that
many git-executables like : 'git config', 'git checkout' 'git add',
'git rev-list', 'git fetch', 'git rev-parse', 'git describe', 'git
hash-object', 'git log' and since these and other shell command comes
from other executables, every command undergoes a through a fork and
exec, thus making the processes slow.

3. Additional dependencies are introduced

When we take a look at the 'git-submodule.sh' file, we can notice that
it is dependent upon various other executables like: 'test', 'printf',
'echo', 'sed', 'grep', 'mkdir', 'rm'. This leads to the requirement of
adding these various dependencies to a non-POSIX environment.

4. Coverage checking is not implemented for shell scripts:

Since currently kcov is not implemented, checking the coverage

Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Daniel Ferreira (theiostream)]

Well, Google requires me to have a draft on a Google Doc anyway for
the proposal, and I am unsure who exactly it will reach. Since it *is*
part of the discussion regarding my proposal, I suppose it is worth
posting here for anyone to comment:
https://docs.google.com/document/d/1dvF2PNRQvvZ351jCdKzOLs7tzaDqhR7ci7TDgzYQg9I/edit?usp=sharing.

-- Daniel.

On Fri, Mar 31, 2017 at 2:07 AM, Daniel Ferreira (theiostream)
<bnm...@gmail.com> wrote:
> Hi Stefan & Johannes,
>
> Thank you for the precious feedback on the proposal. I don't see much
> sense in sending a full "v2" of it and have you read it all over
> again, so I'll just answer to your comments directly.
>
> Also, although the GSoC website allows me to send a "proposal draft"
> to you through the website, since I've already sent it here that
> shouldn't be necessary, correct? I intend to use it just to send the
> final thing.
>
> On Wed, Mar 29, 2017 at 9:01 PM, Johannes Schindelin
> <johannes.schinde...@gmx.de> wrote:
>> On Tue, 28 Mar 2017, Stefan Beller wrote:
>>
>>> On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)
>>> <bnm...@gmail.com> wrote:
>>>
>>> > SYNOPSIS
>>> > There are many advantages to converting parts of git that are still
>>> > scripts to C builtins, among which execution speed, improved
>>> > compatibility and code deduplication.
>>>
>>> agreed.
>>
>> I would even add portability. But yeah, speed is a big thing. I am an
>> extensive user of `git add -p` (which is backed by
>> git-add--interactive.perl) and it is slow as molasses on Windows, just
>> because it is a Perl script (and the Perl interpreter needs to emulate
>> POSIX functionality that is frequently not even needed, such as: copying
>> all memory and reopening all file descriptors in a fork() call only to
>> exec() git.exe right away, tossing all of the diligently work into the
>> dustbin).
>
> Thanks for this example – it hadn't come to my mind since I don't use
> Git on Windows. I'll be sure to complement the synopsis with it. :)
>
>>
>>> > FEASIBILITY
>>> >
>>> > There was only one discussion regarding the feasibility of its porting
>>> > (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1hl...@mail.gmail.com/).
>>> > It resulted in a consensus that doing it would be a task too large –
>>> > although interesting – for GSoC 2015 based on the amount of its lines
>>> > of code. It is, however, only a few lines larger than
>>> > git-rebase--interactive, which has been considered an appropriate
>>> > idea. As such, it looks like a possible project for three months of
>>> > full-time work.
>>>
>>> ok, it sounds a challenging project. (currently counting 1750 lines of
>>> code). Scrolling over the source code, there are quite a couple of
>>> functions, where the direct equivalent in C springs to mind.
>>>
>>> run_cmd_pipe -> see run-command.h
>>> unquote_path -> unquote_c_style ?
>>> refresh -> update_index_if_able()
>>> list_modified -> iterate over "const struct cache_entry *ce = 
>>> active_cache[i];"
>
> Thank you for these functions. I don't think I will be able to specify
> them in detail as part of the projected timeline (e.g. "June 1:
> convert calls to refresh() to use update_index_if_able()") already
> because there is not enough time prior to the proposal deadline to
> study their behavior in detail, and I like to avoid talking about
> things I don't fully understand. Although I think I can cite them as
> examples for a thesis I had put elsewhere in the proposal that "Git
> APIs in Perl already have functional equivalents in C".
>
> Also, they will be great for my early investigation stage into
> git-add--interactive. :) Once more, thanks for having listed them.
>
>> Yes, I think it would be more important to acquaint oneself with the
>> idiosynchracies of Git's internal "API" than to get familiar with Perl:
>> interpreting what obscure Perl code does is something I would gladly do as
>> a mentor.
>
> That's really nice! I usually don't get stuck when trying to
> understand code in languages I'm not too well acquainted with, but I
> figured getting more familiar with Perl would speed development up.
> But it does make sense that this "prior to May 4" might be better
> invested learning about git's internals than Perl.
>
> Question: do you suggest any pending bugfix to git-add--interactive or
> to something related t

Business Proposal

[Mr Youichi Kanno]

Dear Sir/Madam,

My name is Youichi Kanno and I work in Audit & credit Supervisory role at The 
Norinchukin Bank,I am contacting you regarding the asset of a deceased client 
Mr. Grigor Kassan and I need your assistance to process the fund claims oF 
$18,100,000.00 (Eighteen Million, One Hundred Thousand, USD) I would appreciate 
it. I hope to hear from you soon so we can discuss the logistic of moving the 
funds to a safe offshore bank.

Yours sincerely,
Youichi Kanno
Phone Number: +81345400962

Business Proposal

[Mr Youichi Kanno]

Dear Sir/Madam,

My name is Youichi Kanno and I work in Audit & credit Supervisory role at The 
Norinchukin Bank,I am contacting you regarding the asset of a deceased client 
Mr. Grigor Kassan and I need your assistance to process the fund claims oF 
$18,100,000.00 (Eighteen Million, One Hundred Thousand, USD) I would appreciate 
it. I hope to hear from you soon so we can discuss the logistic of moving the 
funds to a safe offshore bank.

Yours sincerely,
Youichi Kanno
Phone Number: +81345400962

Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Daniel Ferreira (theiostream)]

Hi Stefan & Johannes,

Thank you for the precious feedback on the proposal. I don't see much
sense in sending a full "v2" of it and have you read it all over
again, so I'll just answer to your comments directly.

Also, although the GSoC website allows me to send a "proposal draft"
to you through the website, since I've already sent it here that
shouldn't be necessary, correct? I intend to use it just to send the
final thing.

On Wed, Mar 29, 2017 at 9:01 PM, Johannes Schindelin
<johannes.schinde...@gmx.de> wrote:
> On Tue, 28 Mar 2017, Stefan Beller wrote:
>
>> On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)
>> <bnm...@gmail.com> wrote:
>>
>> > SYNOPSIS
>> > There are many advantages to converting parts of git that are still
>> > scripts to C builtins, among which execution speed, improved
>> > compatibility and code deduplication.
>>
>> agreed.
>
> I would even add portability. But yeah, speed is a big thing. I am an
> extensive user of `git add -p` (which is backed by
> git-add--interactive.perl) and it is slow as molasses on Windows, just
> because it is a Perl script (and the Perl interpreter needs to emulate
> POSIX functionality that is frequently not even needed, such as: copying
> all memory and reopening all file descriptors in a fork() call only to
> exec() git.exe right away, tossing all of the diligently work into the
> dustbin).

Thanks for this example – it hadn't come to my mind since I don't use
Git on Windows. I'll be sure to complement the synopsis with it. :)

>
>> > FEASIBILITY
>> >
>> > There was only one discussion regarding the feasibility of its porting
>> > (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1hl...@mail.gmail.com/).
>> > It resulted in a consensus that doing it would be a task too large –
>> > although interesting – for GSoC 2015 based on the amount of its lines
>> > of code. It is, however, only a few lines larger than
>> > git-rebase--interactive, which has been considered an appropriate
>> > idea. As such, it looks like a possible project for three months of
>> > full-time work.
>>
>> ok, it sounds a challenging project. (currently counting 1750 lines of
>> code). Scrolling over the source code, there are quite a couple of
>> functions, where the direct equivalent in C springs to mind.
>>
>> run_cmd_pipe -> see run-command.h
>> unquote_path -> unquote_c_style ?
>> refresh -> update_index_if_able()
>> list_modified -> iterate over "const struct cache_entry *ce = 
>> active_cache[i];"

Thank you for these functions. I don't think I will be able to specify
them in detail as part of the projected timeline (e.g. "June 1:
convert calls to refresh() to use update_index_if_able()") already
because there is not enough time prior to the proposal deadline to
study their behavior in detail, and I like to avoid talking about
things I don't fully understand. Although I think I can cite them as
examples for a thesis I had put elsewhere in the proposal that "Git
APIs in Perl already have functional equivalents in C".

Also, they will be great for my early investigation stage into
git-add--interactive. :) Once more, thanks for having listed them.

> Yes, I think it would be more important to acquaint oneself with the
> idiosynchracies of Git's internal "API" than to get familiar with Perl:
> interpreting what obscure Perl code does is something I would gladly do as
> a mentor.

That's really nice! I usually don't get stuck when trying to
understand code in languages I'm not too well acquainted with, but I
figured getting more familiar with Perl would speed development up.
But it does make sense that this "prior to May 4" might be better
invested learning about git's internals than Perl.

Question: do you suggest any pending bugfix to git-add--interactive or
to something related that might give some useful knowledge in advance?
(for the pre-code period). My microproject involves playing with the
dir_iterator interface, which is a great exercise in code refactoring
but really does not teach me too much about Git's architecture.

Even if you do not have an answer to this, I'm pretty sure I'll keep
this commitment to submitting some patch series somehow related to
git-add before GSoC begins, especially after this comment from
Johannes.

>
>> > PROJECTED TIMELINE
>> > - Prior to May 4
>> > -- Refine my basic knowledge of Perl
>> > -- Craft one or two small patches to some of Git's Perl components
>> > (preferentially to git-add--interactive itself) to improve my
>> > understanding of the language and of how Git's Perl scripts act

Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Johannes Schindelin]

Hi Stefan & Daniel,

On Tue, 28 Mar 2017, Stefan Beller wrote:

> On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)
>  wrote:
> 
> > SYNOPSIS
> > There are many advantages to converting parts of git that are still
> > scripts to C builtins, among which execution speed, improved
> > compatibility and code deduplication.
> 
> agreed.

I would even add portability. But yeah, speed is a big thing. I am an
extensive user of `git add -p` (which is backed by
git-add--interactive.perl) and it is slow as molasses on Windows, just
because it is a Perl script (and the Perl interpreter needs to emulate
POSIX functionality that is frequently not even needed, such as: copying
all memory and reopening all file descriptors in a fork() call only to
exec() git.exe right away, tossing all of the diligently work into the
dustbin).

> > git-add--interactive, one of the most useful features of Git.
> 
> knee jerk reaction: I never used it, so it cannot be that important ;)
> (I use git-gui, which is essentially the same workflow. There are tons
> of ways to accomplish a given goal using Git, so I guess we don't
> want to get in an argument here).

Well, I make up for your lack of `git add -i` usage.

Of course, since you use git-gui, you are simply using another dependency
that bloats Git for Windows: Tcl/Tk.

> > FEASIBILITY
> >
> > There was only one discussion regarding the feasibility of its porting
> > (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1hl...@mail.gmail.com/).
> > It resulted in a consensus that doing it would be a task too large –
> > although interesting – for GSoC 2015 based on the amount of its lines
> > of code. It is, however, only a few lines larger than
> > git-rebase--interactive, which has been considered an appropriate
> > idea. As such, it looks like a possible project for three months of
> > full-time work.
> 
> ok, it sounds a challenging project. (currently counting 1750 lines of
> code). Scrolling over the source code, there are quite a couple of
> functions, where the direct equivalent in C springs to mind.
> 
> run_cmd_pipe -> see run-command.h
> unquote_path -> unquote_c_style ?
> refresh -> update_index_if_able()
> list_modified -> iterate over "const struct cache_entry *ce = 
> active_cache[i];"

Yes, I think it would be more important to acquaint oneself with the
idiosynchracies of Git's internal "API" than to get familiar with Perl:
interpreting what obscure Perl code does is something I would gladly do as
a mentor.

> > PROJECTED TIMELINE
> > - Prior to May 4
> > -- Refine my basic knowledge of Perl
> > -- Craft one or two small patches to some of Git's Perl components
> > (preferentially to git-add--interactive itself) to improve my
> > understanding of the language and of how Git's Perl scripts actually
> > work

As I mentioned above, the Perl code should be fairly intuitive for the
most part, with maybe a couple of pieces of code using more advanced Perl
techniques. Given the scope of the project, I would recommend working
closely together with the mentor(s) to clarify what those code parts do.

> > - May 4 - May 30
> > -- Clarify implementation details with my mentor, and work on a more
> > detailed roadmap for the project
> > -- Investigate roughly how to replace command invocations from the
> > script with actual builtin functions; which Git APIs in Perl already
> > have functional equivalents in C; which parts will require a full
> > rewrite.
> 
> There are different approaches for replacing functionality in another
> language. Examples:
> * Implement the functionality in C and then have a "flag-day" commit
>   783d7e865e (builtin-am: remove redirection to git-am.sh, 2015-08-04)
>   This only works when the whole functionality was replaced in prior commits
> * Implement partial functionality in C and call it via a helper function.
>   3604242f08 (submodule: port init from shell to C, 2016-04-15)
>   This works well for only partial conversions (the larger the thing to
>   convert the more appealing this is, as it gets code shipped early.)
>   When choosing this strategy, this part of the Project would be to
>   identify parts that could be ported on its own without much
>   additional glue-code.

To offer my perspective: I strongly prefer the latter approach. Not only
does it yield earlier results, it also makes it substantially easier to
handle the project even if it should turn out to be a little larger than
just 3 months.

> > - May 30 - June 30 (start of coding period)
> > -- Define the architecture of the builtin within git (which
> > functions/interfaces will it have? where will its code reside?).
> > -- Implement a small subset of the builtin (to be defined with my
> > mentor) and glue it into the existing Perl script. Present this as a
> > first patch to get feedback early regarding the implementation and
> > avoid piling up mistakes early.
> > -- Do necessary changes based on this initial review.
> > -- Have 

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Stefan Beller]

+cc Ben Peart, who sent
"[RFC] Add support for downloading blobs on demand" to the list recently.
This proposal here seems like it has the same goal, so maybe your review
could go a long way here?

Thanks,
Stefan

On Tue, Mar 14, 2017 at 3:57 PM, Jonathan Tan <jonathanta...@google.com> wrote:
> As described in "Background" below, there have been at least 2 patch sets to
> support "partial clones" and on-demand blob fetches, where the server part
> that supports on-demand blob fetches was treated at least in outline. Here
> is a proposal treating that server part in detail.
>
> == Background
>
> The desire for Git to support (i) missing blobs and (ii) fetching them as
> needed from a remote repository has surfaced on the mailing list a few
> times, most recently in the form of RFC patch sets [1] [2].
>
> A local repository that supports (i) will be created by a "partial clone",
> that is, a clone with some special parameters (exact parameters are still
> being discussed) that does not download all blobs normally downloaded. Such
> a repository should support (ii), which is what this proposal describes.
>
> == Design
>
> A new endpoint "server" is created. The client will send a message in the
> following format:
>
> 
> fbp-request = PKT-LINE("fetch-blob-pack")
>   1*want
>   flush-pkt
> want = PKT-LINE("want" SP obj-id)
> 
>
> The client may send one or more SHA-1s for which it wants blobs, then a
> flush-pkt.
>
> The server will then reply:
>
> 
> server-reply = flush-pkt | PKT-LINE("ERR" SP message)
> 
>
> If there was no error, the server will then send them in a packfile,
> formatted like described in "Packfile Data" in pack-protocol.txt with
> "side-band-64k" enabled.
>
> Any server that supports "partial clone" will also support this, and the
> client will automatically assume this. (How a client discovers "partial
> clone" is not covered by this proposal.)
>
> The server will perform reachability checks on requested blobs through the
> equivalent of "git rev-list --use-bitmap-index" (like "git upload-pack" when
> using the allowreachablesha1inwant option), unless configured to suppress
> reachability checks through a config option. The server administrator is
> highly recommended to regularly regenerate the bitmap (or suppress
> reachability checks).
>
> === Endpoint support for forward compatibility
>
> This "server" endpoint requires that the first line be understood, but will
> ignore any other lines starting with words that it does not understand. This
> allows new "commands" to be added (distinguished by their first lines) and
> existing commands to be "upgraded" with backwards compatibility.
>
> === Related improvements possible with new endpoint
>
> Previous protocol upgrade suggestions have had to face the difficulty of
> allowing updated clients to discover the server support while not slowing
> down (for example, through extra network round-trips) any client, whether
> non-updated or updated. The introduction of "partial clone" allows clients
> to rely on the guarantee that any server that supports "partial clone"
> supports "fetch-blob-pack", and we can extend the guarantee to other
> protocol upgrades that such repos would want.
>
> One such upgrade is "ref-in-want" [3]. The full details can be obtained from
> that email thread, but to summarize, the patch set eliminates the need for
> the initial ref advertisement and allows communication in ref name globs,
> making it much easier for multiple load-balanced servers to serve large
> repos to clients - this is something that would greatly benefit the Android
> project, for example, and possibly many others.
>
> Bundling support for "ref-in-want" with "fetch-blob-pack" simplifies matters
> for the client in that a client needs to only handle one "version" of server
> (a server that supports both). If "ref-in-want" were added later, instead of
> now, clients would need to be able to handle two "versions" (one with only
> "fetch-blob-pack" and one with both "fetch-blob-pack" and "ref-in-want").
>
> As for its implementation, that email thread already contains a patch set
> that makes it work with the existing "upload-pack" endpoint; I can update
> that patch set to use the proposed "server" endpoint (with a
> "fetch-commit-pack" message) if need be.
>
> == Client behavior
>
> This proposal is concerned with server b

Re: [GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Stefan Beller]

On Sat, Mar 25, 2017 at 8:15 PM, Daniel Ferreira (theiostream)
 wrote:

> SYNOPSIS
> There are many advantages to converting parts of git that are still
> scripts to C builtins, among which execution speed, improved
> compatibility and code deduplication.

agreed.

> git-add--interactive, one of the most useful features of Git.

knee jerk reaction: I never used it, so it cannot be that important ;)
(I use git-gui, which is essentially the same workflow. There are tons
of ways to accomplish a given goal using Git, so I guess we don't
want to get in an argument here).

>
> FEASIBILITY
>
> There was only one discussion regarding the feasibility of its porting
> (https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1hl...@mail.gmail.com/).
> It resulted in a consensus that doing it would be a task too large –
> although interesting – for GSoC 2015 based on the amount of its lines
> of code. It is, however, only a few lines larger than
> git-rebase--interactive, which has been considered an appropriate
> idea. As such, it looks like a possible project for three months of
> full-time work.

ok, it sounds a challenging project. (currently counting 1750 lines of
code). Scrolling over the source code, there are quite a couple of
functions, where the direct equivalent in C springs to mind.

run_cmd_pipe -> see run-command.h
unquote_path -> unquote_c_style ?
refresh -> update_index_if_able()
list_modified -> iterate over "const struct cache_entry *ce = active_cache[i];"


> PROJECTED TIMELINE
> - Prior to May 4
> -- Refine my basic knowledge of Perl
> -- Craft one or two small patches to some of Git's Perl components
> (preferentially to git-add--interactive itself) to improve my
> understanding of the language and of how Git's Perl scripts actually
> work



>
> - May 4 - May 30
> -- Clarify implementation details with my mentor, and work on a more
> detailed roadmap for the project
> -- Investigate roughly how to replace command invocations from the
> script with actual builtin functions; which Git APIs in Perl already
> have functional equivalents in C; which parts will require a full
> rewrite.

There are different approaches for replacing functionality in another
language. Examples:
* Implement the functionality in C and then have a "flag-day" commit
  783d7e865e (builtin-am: remove redirection to git-am.sh, 2015-08-04)
  This only works when the whole functionality was replaced in prior commits
* Implement partial functionality in C and call it via a helper function.
  3604242f08 (submodule: port init from shell to C, 2016-04-15)
  This works well for only partial conversions (the larger the thing to
  convert the more appealing this is, as it gets code shipped early.)
  When choosing this strategy, this part of the Project would be to
  identify parts that could be ported on its own without much
  additional glue-code.

> - May 30 - June 30 (start of coding period)
> -- Define the architecture of the builtin within git (which
> functions/interfaces will it have? where will its code reside?).
> -- Implement a small subset of the builtin (to be defined with my
> mentor) and glue it into the existing Perl script. Present this as a
> first patch to get feedback early regarding the implementation and
> avoid piling up mistakes early.
> -- Do necessary changes based on this initial review.
> -- Have roughly 1/3 of the script's functionality ported to C.
>
> - June 30 - July 28
> -- Port the remainder of the script to a builtin.
> -- Have a weekly roadmap, sending a part of the patch every 15 days to
> the mailing list for review and to avoid massive commits by the end of
> GSoC.

yeah; send early, send often. ;)

> -- Apply suggestions from community reviews when possible; if not,
> save them for doing toward the end of GSoC (see below).

Please do not underestimate the discussion by community, finding
consensus on list consumes a bit of time in some cases.

> (Note: due to a previous commitment, during a five-day period of July
> I will only be able to work part-time on GSoC. The actual week will be
> known over the next weeks.)

Maybe you want to shift the schedule up to here by one week then?
(e.g. the first period would be April 27  - May 23)

>
> - July 28 - August 29
> -- By the start of this period, send a patch with the builtin fully
> implemented to the mailing list.

/a patch/a patch series consisting of many patches/
Experience shows that smaller patches are easier to review as
it is more focused. Consider e.g. e86ab2c1cd (wt-status: convert
to struct object_id, 2017-02-21) and the parents leading up to this
commit. They work on the same big topic, but focus on very
regional areas to ease review.

> -- Fix bugs, test extensively, possibly extend test coverage for
> git-add--interactive.

AFAICT ('$ git grep "git add -i"') there is only t3701 testing the
interactive add. Maybe we need to add tests first to document
current behavior, before attempting a conversion?

Re: [GSoC] Proposal Discussion

[Devin Lehmacher]

On Tue, Mar 28, 2017 at 07:52:42AM +0200, Christian Couder wrote:
> Hi,
>
> On Tue, Mar 28, 2017 at 12:17 AM, Devin Lehmacher <lehma...@gmail.com> wrote:
> > Hello everyone,
> >
> > I am a student studying Computer Science at Cornell University. I
> > already completed a microproject, Move ~/.git-credential-cache/socket to
> > $XDG_CACHE_HOME/credential/socket a week and a half ago or so.
>
> Nice. It would be better though if you could provide a link to the
> thread where your microproject was discussed. If it has been merged to
> master, you could also provide the merge commit. Otherwise please tell
> what is its branch name and current status in the last "What's cooking
> in git.git" email from Junio.

Here is the merge commit into master: 78cf8efec34c419ecea86bc8d1fe47ec0b51ba37

> > I am interested in 2 different projects and would like some advice on
> > them, to help me decide which one to submit a proposal for.
> >
> > 1. `git rebase -i` conversion.
> >I was initially the most interested in this project but realize that
> >after having a very busy week last week that Ivan Tham started
> >[discussion][1] about this project. Would it be appropriate to submit
> >    a proposal for a project that someone else also wants to work on?
>
> Yes, it is ok. Obviously only one student/proposal can be selected for
> a given project, but as anyway the main constraint for us is usually
> the number of available mentors, there is a low chance that this would
> prevent us from selecting one more student than we could otherwise
> select.
>
> You could also submit 2 proposals if you have time to work on more than one.

Ok! I think I will post rough drafts of both proposals sometime tomorrow
or maybe later today if I have time.

> > 2. formatting tool improvements.
> >There are four different git commands mentioned [here][2] as possible
> >tools to improve as can be seen in the email. Of those I think it
> >would make the most sense to extend `git name-rev`. It seems best
> >suited to the desired behavior. It would need to be extended to
> >understand rev's that refer to objects rather than just a commit-ish
> >and also add formatting support similar to the information that log
> >and for-each-ref can output. Since this doesn't seem like much work,
> >would it be feasible to generalize and somewhat standardize all of
> >the formatting commands?
>
> Yeah, I think it would be good. It might involve a lot of discussion
> though and this could slow your project.
> So if you really want to do it, my advice is to try to start the
> discussion as soon as possible, that is now.
>
> To do that you could for example Cc people involved in the email
> discussions, and try to come up with concrete proposals about how to
> generalize and standardize the formatting commands.

I will try to send out an email later this afternoon with a preliminary
plan and to start discussion about how best to rework formatting
commands.

Thanks,
Devin

Re: [GSoC] Proposal Discussion

[Christian Couder]

Hi,

On Tue, Mar 28, 2017 at 12:17 AM, Devin Lehmacher <lehma...@gmail.com> wrote:
> Hello everyone,
>
> I am a student studying Computer Science at Cornell University. I
> already completed a microproject, Move ~/.git-credential-cache/socket to
> $XDG_CACHE_HOME/credential/socket a week and a half ago or so.

Nice. It would be better though if you could provide a link to the
thread where your microproject was discussed. If it has been merged to
master, you could also provide the merge commit. Otherwise please tell
what is its branch name and current status in the last "What's cooking
in git.git" email from Junio.

> I am interested in 2 different projects and would like some advice on
> them, to help me decide which one to submit a proposal for.
>
> 1. `git rebase -i` conversion.
>I was initially the most interested in this project but realize that
>after having a very busy week last week that Ivan Tham started
>[discussion][1] about this project. Would it be appropriate to submit
>a proposal for a project that someone else also wants to work on?

Yes, it is ok. Obviously only one student/proposal can be selected for
a given project, but as anyway the main constraint for us is usually
the number of available mentors, there is a low chance that this would
prevent us from selecting one more student than we could otherwise
select.

You could also submit 2 proposals if you have time to work on more than one.

> 2. formatting tool improvements.
>There are four different git commands mentioned [here][2] as possible
>tools to improve as can be seen in the email. Of those I think it
>would make the most sense to extend `git name-rev`. It seems best
>suited to the desired behavior. It would need to be extended to
>understand rev's that refer to objects rather than just a commit-ish
>and also add formatting support similar to the information that log
>and for-each-ref can output. Since this doesn't seem like much work,
>would it be feasible to generalize and somewhat standardize all of
>the formatting commands?

Yeah, I think it would be good. It might involve a lot of discussion
though and this could slow your project.
So if you really want to do it, my advice is to try to start the
discussion as soon as possible, that is now.

To do that you could for example Cc people involved in the email
discussions, and try to come up with concrete proposals about how to
generalize and standardize the formatting commands.

Thanks,
Christian.

[GSoC] Proposal Discussion

[Devin Lehmacher]

Hello everyone,

I am a student studying Computer Science at Cornell University. I
already completed a microproject, Move ~/.git-credential-cache/socket to
$XDG_CACHE_HOME/credential/socket a week and a half ago or so.

I am interested in 2 different projects and would like some advice on
them, to help me decide which one to submit a proposal for.

1. `git rebase -i` conversion. 
   I was initially the most interested in this project but realize that
   after having a very busy week last week that Ivan Tham started
   [discussion][1] about this project. Would it be appropriate to submit
   a proposal for a project that someone else also wants to work on?

2. formatting tool improvements. 
   There are four different git commands mentioned [here][2] as possible
   tools to improve as can be seen in the email. Of those I think it
   would make the most sense to extend `git name-rev`. It seems best
   suited to the desired behavior. It would need to be extended to
   understand rev's that refer to objects rather than just a commit-ish
   and also add formatting support similar to the information that log
   and for-each-ref can output. Since this doesn't seem like much work,
   would it be feasible to generalize and somewhat standardize all of
   the formatting commands?

[1]: 
https://public-inbox.org/git/20170320164154.xbcu6rg0c%25pickf...@riseup.net/
[2]: 
https://public-inbox.org/git/CA+P7+xr4ZNCCJkS0=yR-FNu+MrL60YX-+Wsz9L_5LCNhnY_d=a...@mail.gmail.com/

Thanks for any feedback you may have,
Devin

[GSoC] Proposal: turn git-add--interactive.perl into a builtin

[Daniel Ferreira (theiostream)]

Hi there. First of all, I'd like to thank all of the support up to now
with my microproject :). Here's a first draft of my proposal for
Google Summer of Code '17, based on the "Convert scripts to builtins"
idea. Please let me know what you think.

---

SYNOPSIS
There are many advantages to converting parts of git that are still
scripts to C builtins, among which execution speed, improved
compatibility and code deduplication. This proposal aims to apply this
to git-add--interactive, one of the most useful features of Git.

FEASIBILITY
Many git scripts have attracted attention for being turned into
builtins. There is ongoing work on git-stash
(https://public-inbox.org/git/20170321053135.thk77soxc4irx...@sigill.intra.peff.net/),
and porting interactive rebase is one of the ideas for this edition of
GSoC. Not as much attention, however, has been directed to
git-add--interactive.

There was only one discussion regarding the feasibility of its porting
(https://public-inbox.org/git/CAP8UFD2PcBsU6=FK4OHVrB7E98ycohS_0pYcbCBar=of1hl...@mail.gmail.com/).
It resulted in a consensus that doing it would be a task too large –
although interesting – for GSoC 2015 based on the amount of its lines
of code. It is, however, only a few lines larger than
git-rebase--interactive, which has been considered an appropriate
idea. As such, it looks like a possible project for three months of
full-time work.

Aside from the benefits cited above, turning git-add--interactive into
a builtin can reduce Git's dependency on Perl to the point where no
"common" command would continue to rely on it.

PROJECTED TIMELINE
- Prior to May 4
-- Refine my basic knowledge of Perl
-- Craft one or two small patches to some of Git's Perl components
(preferentially to git-add--interactive itself) to improve my
understanding of the language and of how Git's Perl scripts actually
work

- May 4 - May 30
-- Clarify implementation details with my mentor, and work on a more
detailed roadmap for the project
-- Investigate roughly how to replace command invocations from the
script with actual builtin functions; which Git APIs in Perl already
have functional equivalents in C; which parts will require a full
rewrite.

- May 30 - June 30 (start of coding period)
-- Define the architecture of the builtin within git (which
functions/interfaces will it have? where will its code reside?).
-- Implement a small subset of the builtin (to be defined with my
mentor) and glue it into the existing Perl script. Present this as a
first patch to get feedback early regarding the implementation and
avoid piling up mistakes early.
-- Do necessary changes based on this initial review.
-- Have roughly 1/3 of the script's functionality ported to C.

- June 30 - July 28
-- Port the remainder of the script to a builtin.
-- Have a weekly roadmap, sending a part of the patch every 15 days to
the mailing list for review and to avoid massive commits by the end of
GSoC.
-- Apply suggestions from community reviews when possible; if not,
save them for doing toward the end of GSoC (see below).
(Note: due to a previous commitment, during a five-day period of July
I will only be able to work part-time on GSoC. The actual week will be
known over the next weeks.)

- July 28 - August 29
-- By the start of this period, send a patch with the builtin fully
implemented to the mailing list.
-- Fix bugs, test extensively, possibly extend test coverage for
git-add--interactive.
-- Respond to the (predictably big) community feedback regarding the change.

I currently work full-time in a payments company (see below), but in
case of being accepted I am willing to quit my job some months early
to dedicate myself fully to GSoC starting June.

BIOGRAPHICAL INFORMATION
My name is Daniel Ferreira and I'm a student from São Paulo, Brazil. I
was accepted by Stanford University last year and I will start college
this fall. I started coding C about six years ago writing up system
modifications ("tweaks") for jailbroken iPhones. Since then, I have
written/contributed to a couple of open-source projects like an IRC
bot and other assorted things – all of them tracked on Git
(https://github.com/theiostream). I have also developed a
(closed-source) library in C for interacting with payment terminals in
the company I have worked for over the last two years (Pagar.me).
There, we use Git extensively for managing projects with around 20
people working concurrently.

MICROPROJECT
I have sent a series of patches to complete the microproject of
converting recursive calls to readdir() into calls to dir_iterator.
The most recent version can be found in
https://public-inbox.org/git/1490465551-71056-2-git-send-email-bnm...@gmail.com/T/#u.

Thanks,
-- Daniel.

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Jonathan Tan]

On 03/16/2017 02:17 PM, Junio C Hamano wrote:

Yeah, the example was solely to see how the system was to be
extended, as one of the selling point of the proposal was:

> === Endpoint support for forward compatibility
>
> This "server" endpoint requires that the first line be understood, but
> will ignore any other lines starting with words that it does not
> understand. This allows new "commands" to be added (distinguished by
> their first lines) and existing commands to be "upgraded" with
> backwards compatibility.






And the lack of "capability negotiation" is substituted by "assume
the better server, fallback to lower common denominator by detecting
errors"?


Yes. I probably should have mentioned that this "forward compatibility" 
is limited - it does not include any potential new feature intending to 
reduce the size of the request. I was thinking more of this being able 
to be extended to, for example, add "hint" lines that certain blobs come 
from certain commits, or add "have" lines to present blobs that might be 
good delta bases (and even if the server doesn't understand these lines, 
its output is still correct).

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Junio C Hamano]

Jonathan Tan <jonathanta...@google.com> writes:

> On 03/15/2017 10:59 AM, Junio C Hamano wrote:
>> ...
>> but I am wondering how you would extend the proposed system to do
>> so.  Would you add "fetch-size-limited-blob-in-tree-pack" that runs
>> parallel to "fetch-blob-pack" request?  Would you add a new type of
>> request packet "want-blob-with-expression" for fbp-request, which is
>> protected by some "protocol capability" exchange?
>>
>> If the former, how does a client discover if a particular server
>> already supports the new "fetch-size-limited-blob-in-tree-pack"
>> request, so that it does not have to send a bunch of "want" request
>> by enumerating the blobs itself?  If the latter, how does a client
>> discover if a particular server's "fetch-blob-pack" already supports
>> the new "want-blob-with-expression" request packet?
>
> I'm not sure if that use case is something we need to worry about (if
> you're downloading x * 10MB, uploading x * 50B shouldn't be a problem,
> I think), but if we want to handle that use case in the future, I
> agree that extending this system would be difficult.

Yeah, the example was solely to see how the system was to be
extended, as one of the selling point of the proposal was:

> === Endpoint support for forward compatibility
>
> This "server" endpoint requires that the first line be understood, but
> will ignore any other lines starting with words that it does not
> understand. This allows new "commands" to be added (distinguished by
> their first lines) and existing commands to be "upgraded" with
> backwards compatibility.

> The best way I can think of right now is for the client to send a
> fetch-blob-pack request with no "want" lines and at least one
> "want-tree" line, ...

So it is not by adding new type of "request" that sits next to
"fetch-blob-pack" request, but by adding a new way to drive that
existing "fetch-blob-pack" request.  

> and then if there is an error (which will happen if
> the server is old, and therefore sees that there is not at least
> "want" line), to retry with the "want" lines. This allows us to add
> alternative ways of specifying blobs later (if we want to), but also
> means that upgrading a client without upgrading the corresponding
> server incurs a round-trip penalty.

And the lack of "capability negotiation" is substituted by "assume
the better server, fallback to lower common denominator by detecting
errors"?

> Alternatively we could add rudimentary support for trees now and add
> filter-by-size later ...

I am not particularly interested in "blobs in this tree" request.
It was merely an example to make it easier to discuss the main
point, which is the bigger picture that the proposal was painting
around "forward compatibility".

Thanks.

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Jonathan Tan]

On 03/15/2017 10:59 AM, Junio C Hamano wrote:

By "SHA-1s for which it wants blobs", you mean that "want" only
allows one exact blob object name?  I think it is necessary to
support that mode of operation as a base case, and it is a good
starting point.

When you know

 - you have a "partial" clone that initially asked to contain only
   blobs that are smaller than 10MB, and

 - you are now trying to do a "git checkout v1.0 -- this/directory"
   so that the directory is fully populated

instead of enumerating all the missing blobs from the output of
"ls-tree -r v1.0 this/directory" on separate "want" requests, you
may want to say "I want all the blobs that are not smaller than 10MB
in this tree object $(git rev-parse v1.0:this/directory)".

I am not saying that you should add something like this right away,
but I am wondering how you would extend the proposed system to do
so.  Would you add "fetch-size-limited-blob-in-tree-pack" that runs
parallel to "fetch-blob-pack" request?  Would you add a new type of
request packet "want-blob-with-expression" for fbp-request, which is
protected by some "protocol capability" exchange?

If the former, how does a client discover if a particular server
already supports the new "fetch-size-limited-blob-in-tree-pack"
request, so that it does not have to send a bunch of "want" request
by enumerating the blobs itself?  If the latter, how does a client
discover if a particular server's "fetch-blob-pack" already supports
the new "want-blob-with-expression" request packet?


I'm not sure if that use case is something we need to worry about (if 
you're downloading x * 10MB, uploading x * 50B shouldn't be a problem, I 
think), but if we want to handle that use case in the future, I agree 
that extending this system would be difficult.


The best way I can think of right now is for the client to send a 
fetch-blob-pack request with no "want" lines and at least one 
"want-tree" line, and then if there is an error (which will happen if 
the server is old, and therefore sees that there is not at least "want" 
line), to retry with the "want" lines. This allows us to add alternative 
ways of specifying blobs later (if we want to), but also means that 
upgrading a client without upgrading the corresponding server incurs a 
round-trip penalty.


Alternatively we could add rudimentary support for trees now and add 
filter-by-size later (so that such requests made to old servers will 
download extra blobs, but at least it works), but it still doesn't solve 
the general problem of specifying blobs by some other rule than its own 
SHA-1 or its tree's SHA-1.

Re: Proposal for "fetch-any-blob Git protocol" and server design

[Junio C Hamano]

Jonathan Tan  writes:

> == Design
>
> A new endpoint "server" is created. The client will send a message in
> the following format:
>
> 
> fbp-request = PKT-LINE("fetch-blob-pack")
>   1*want
>   flush-pkt
> want = PKT-LINE("want" SP obj-id)
> 
>
> The client may send one or more SHA-1s for which it wants blobs, then
> a flush-pkt.

By "SHA-1s for which it wants blobs", you mean that "want" only
allows one exact blob object name?  I think it is necessary to
support that mode of operation as a base case, and it is a good
starting point.

When you know

 - you have a "partial" clone that initially asked to contain only
   blobs that are smaller than 10MB, and

 - you are now trying to do a "git checkout v1.0 -- this/directory"
   so that the directory is fully populated

instead of enumerating all the missing blobs from the output of
"ls-tree -r v1.0 this/directory" on separate "want" requests, you
may want to say "I want all the blobs that are not smaller than 10MB
in this tree object $(git rev-parse v1.0:this/directory)".

I am not saying that you should add something like this right away,
but I am wondering how you would extend the proposed system to do
so.  Would you add "fetch-size-limited-blob-in-tree-pack" that runs
parallel to "fetch-blob-pack" request?  Would you add a new type of
request packet "want-blob-with-expression" for fbp-request, which is
protected by some "protocol capability" exchange?

If the former, how does a client discover if a particular server
already supports the new "fetch-size-limited-blob-in-tree-pack"
request, so that it does not have to send a bunch of "want" request
by enumerating the blobs itself?  If the latter, how does a client
discover if a particular server's "fetch-blob-pack" already supports
the new "want-blob-with-expression" request packet?

> === Endpoint support for forward compatibility
>
> This "server" endpoint requires that the first line be understood, but
> will ignore any other lines starting with words that it does not
> understand. This allows new "commands" to be added (distinguished by
> their first lines) and existing commands to be "upgraded" with
> backwards compatibility.

Proposal for "fetch-any-blob Git protocol" and server design

[Jonathan Tan]

As described in "Background" below, there have been at least 2 patch 
sets to support "partial clones" and on-demand blob fetches, where the 
server part that supports on-demand blob fetches was treated at least in 
outline. Here is a proposal treating that server part in detail.


== Background

The desire for Git to support (i) missing blobs and (ii) fetching them 
as needed from a remote repository has surfaced on the mailing list a 
few times, most recently in the form of RFC patch sets [1] [2].


A local repository that supports (i) will be created by a "partial 
clone", that is, a clone with some special parameters (exact parameters 
are still being discussed) that does not download all blobs normally 
downloaded. Such a repository should support (ii), which is what this 
proposal describes.


== Design

A new endpoint "server" is created. The client will send a message in 
the following format:



fbp-request = PKT-LINE("fetch-blob-pack")
  1*want
  flush-pkt
want = PKT-LINE("want" SP obj-id)


The client may send one or more SHA-1s for which it wants blobs, then a 
flush-pkt.


The server will then reply:


server-reply = flush-pkt | PKT-LINE("ERR" SP message)


If there was no error, the server will then send them in a packfile, 
formatted like described in "Packfile Data" in pack-protocol.txt with 
"side-band-64k" enabled.


Any server that supports "partial clone" will also support this, and the 
client will automatically assume this. (How a client discovers "partial 
clone" is not covered by this proposal.)


The server will perform reachability checks on requested blobs through 
the equivalent of "git rev-list --use-bitmap-index" (like "git 
upload-pack" when using the allowreachablesha1inwant option), unless 
configured to suppress reachability checks through a config option. The 
server administrator is highly recommended to regularly regenerate the 
bitmap (or suppress reachability checks).


=== Endpoint support for forward compatibility

This "server" endpoint requires that the first line be understood, but 
will ignore any other lines starting with words that it does not 
understand. This allows new "commands" to be added (distinguished by 
their first lines) and existing commands to be "upgraded" with backwards 
compatibility.


=== Related improvements possible with new endpoint

Previous protocol upgrade suggestions have had to face the difficulty of 
allowing updated clients to discover the server support while not 
slowing down (for example, through extra network round-trips) any 
client, whether non-updated or updated. The introduction of "partial 
clone" allows clients to rely on the guarantee that any server that 
supports "partial clone" supports "fetch-blob-pack", and we can extend 
the guarantee to other protocol upgrades that such repos would want.


One such upgrade is "ref-in-want" [3]. The full details can be obtained 
from that email thread, but to summarize, the patch set eliminates the 
need for the initial ref advertisement and allows communication in ref 
name globs, making it much easier for multiple load-balanced servers to 
serve large repos to clients - this is something that would greatly 
benefit the Android project, for example, and possibly many others.


Bundling support for "ref-in-want" with "fetch-blob-pack" simplifies 
matters for the client in that a client needs to only handle one 
"version" of server (a server that supports both). If "ref-in-want" were 
added later, instead of now, clients would need to be able to handle two 
"versions" (one with only "fetch-blob-pack" and one with both 
"fetch-blob-pack" and "ref-in-want").


As for its implementation, that email thread already contains a patch 
set that makes it work with the existing "upload-pack" endpoint; I can 
update that patch set to use the proposed "server" endpoint (with a 
"fetch-commit-pack" message) if need be.


== Client behavior

This proposal is concerned with server behavior only, but it is useful 
to envision how the client would use this to ensure that the server 
behavior is useful.


=== Indication to use the proposed endpoint

The client will probably already record that at least one of its remotes 
(the one that it successfully performed a "partial clone" from) supports 
this new endpoint (if not, it can’t determine whether a missing blob was 
caused by repo corruption or by the "partial clone"). This knowledge can 
be used both to know that the server supports "fetch-blob-pack" and 
"fetch-commit-pack" (for the latter, the client can fall back to 
"fetch-pack"/"upload-pack"

Business proposal

[Qatif Oil Group Of Companies---]

--
Dear Friend,

I would like to discuss a very important issue with you. I am writing  
to find out if this is your valid email. Please, let me know if this  
email is valid


Kind regards
Adrien Saif
Attorney to Qatif Group of Companies.

Business proposal

[Qatif Oil Group Of Companies---]

--
Dear Friend,

I would like to discuss a very important issue with you. I am writing  
to find out if this is your valid email. Please, let me know if this  
email is valid


Kind regards
Adrien Saif
Attorney to Qatif Group of Companies.

Business proposal

[Qatif Group of Companies>>>>>]

Dear Friend,

I would like to discuss a very important issue with you. I am writing to
find out if this is your valid email. Please, let me know if this email is
valid

Kind regards
Adrien Saif
Attorney to Qatif Group of Companies

Business Proposal

[QUATIF GROUP OF COMPANIES]

Dear Friend,

I would like to discuss a very important issue with you. I am writing to
find out if this is your valid email. Please, let me know if this email is
valid

Kind regards
Adrien Saif
Attorney to Quatif Group of Companies

How To Write Thesis Proposal in Best Way

[raniya]

Writing a winning  Thesis Proposal
<http://www.dissertationhelp.ae/research-proposal.php>   is considered to be
very difficult task by the students.But now you don't need to be worried
about thesis proposal writing  DissertationHelp.ae
<http://www.dissertationhelp.ae/research-proposal.php>   has helped many
students to finish their thesis research with less stress and confidence.





-----
Proposal Writer 


--
View this message in context: 
http://git.661346.n2.nabble.com/How-To-Write-Thesis-Proposal-in-Best-Way-tp7657464.html
Sent from the git mailing list archive at Nabble.com.

Re: Proposal for an increased `gitk` cohesion with `git stash`.

[Uxío Prego]

Dearest all,

am sorry my previous message did not enter the list (cross my fingers this 
will). I won't be pasting it verbatim because shame on me it leaked zombie 
processes (but that part got silently dropped out by kind Paul).

In case anyone could be interested in the topic, and because a thorough reply 
will take me some time, my most recent edit of this is hosted at 
https://gist.githubusercontent.com/uprego/d8c3c059c56ebb911974bb905157a81e/raw/6a08d9e0ce9c2b1decd4ed92acc924961c7f7769/gitk%2520multi%2520stash%2520patch.
 All problems shown I still think is a nice start (of course p.o.c / pre alpha) 
if anyone ever wanted to get this working or even fix the current problems it 
has.

As Paul recommend I'll be reworking and giving a patch against a rev of his 
upstream.

I'm going to try his code tips to improve non obvious design choices, and (even 
he doesn’t commented it and seems to me most important) really put an extra 
effort in not changing the behaviour of `gitk` (i.e. started without '--all').

Then some testing against large repos, github.com/cartodb/cartodb then 
github.com/odoo/oodo finally Linux; will be done.

The performance issue Paul points to, I don't think is impacting me, but now I 
reckon (just as one example) there are people who develop using IDEs that leave 
garbaged unuseful stashes, and that has to be taken into account as scenario. 
And the large repos. But this file event handlers thing is something that will 
make me lag to fix it, even surprised me because the remaining of the 
subroutines that I patched are just doing the same I typed, I just replicated 
near source (general revs processing) because I have no idea Tcl, even do not 
give a shit, but have to say Tcl is fun C: and an interesting discovery though. 
I hope that was not a trick to get me into improving the performance of the 
near loop that process ALL involved revs (and the similar for refs)! I'm old 
and tired to get into performance hacking. 

I guess you know, the underworked grep must be an easy solve, probably 
excluding ' refs/stash' because a branch named 'refs/stash' is allowed but not 
a branch named ' refs/stash' (IDK which version I was trying but I will try 
both 1.x.y and 2.x.y in time).

Finally... if you don't leverage stashing too much, what is the practice? 
committing ephemeral to later reset and recommit? I hope I don't just needed a 
lesson on git-reset instead of all this.

Please pardon my potentially mangler mail client. Yours sincerely, regards and 
thanks for your time,

> On 12 Dec 2016, at 10:36, Paul Mackerras  wrote:
> 
> Hi Uxio,
> 
> On Thu, Sep 08, 2016 at 03:41:29PM +0200, Uxío Prego wrote:
>> Hello, please forgive me for not introducing me.
>> 
>> +---+
>> |Description|
>> +---+
>> 
>> Patch for showing all stashes in `gitk`.
>> 
>> +---+
>> |The problem|
>> +---+
>> 
>> Being `gitk` one of the best tools for navigating and understanding graphs
>> of git repo revs, I got used to have it for home use, some years ago, soon
>> for office use too.
>> 
>> At some point I got used to invoke always `gitk --all` in order to keep
>> tracking of tags when using the program for building, when possible, stable
>> versions of any GNU/Linux software I would want to use.
>> 
>> It seems `gitk --all` uses `git show-ref` for showing remotes information.
>> A side effect of this is that the most recent stash gets shown in `gitk
>> --all`. After learning what the stash was, I got used to it.
>> 
>> Later, when at jobs, I got used to have a stash on all working branch tips.
>> So I often would have several stashes in `git stash list` but only the last
>> one in `gitk --all`. That annoyed me for about a year, finally I got
>> working in `gitk` to show a stash status more similar to what `git stash
>> list` shows.
>> 
>> +---+
>> |The feature|
>> +---+
>> 
>> Show all stashes in `gitk` instead of only the last one.
> 
> This seems like a good feature, although I don't use stashes myself.
> 
>> +--+
>> |Why it's desirable|
>> +--+
>> 
>> In order to have better visual control when working on repos that have
>> several active branches and there are needed quick context changes between
>> them with the features that `git stash [apply [STASHNAME]| list | pop
>> [STASHNAME]| push | save | show [[-p] STASHNAME]]`.
>> 
>> In order to have a better cohesion between the mentioned features and `gitk
>> --all`.
>> 
>> ++
>> |Caveats and side effects|
>> ++
>> 
>> With this patch several side effects happen:
>> 
>> * Stashes are shown even invoking `gitk`, not only when running `gitk
>> --all`. If this is a problem, I can keep working in the patch to avoid this.
>> 
>> * The most recent stash, which was previously shown as 'stash' because its
>> corresponding `git show-ref` output line reads 'refs/stash', gets shown as
>> 'stash@{0}'. Not removing lines with 

Re: Proposal for an increased `gitk` cohesion with `git stash`.

[Paul Mackerras]

Hi Uxio,

On Thu, Sep 08, 2016 at 03:41:29PM +0200, Uxío Prego wrote:
> Hello, please forgive me for not introducing me.
> 
> +---+
> |Description|
> +---+
> 
> Patch for showing all stashes in `gitk`.
> 
> +---+
> |The problem|
> +---+
> 
> Being `gitk` one of the best tools for navigating and understanding graphs
> of git repo revs, I got used to have it for home use, some years ago, soon
> for office use too.
> 
> At some point I got used to invoke always `gitk --all` in order to keep
> tracking of tags when using the program for building, when possible, stable
> versions of any GNU/Linux software I would want to use.
> 
> It seems `gitk --all` uses `git show-ref` for showing remotes information.
> A side effect of this is that the most recent stash gets shown in `gitk
> --all`. After learning what the stash was, I got used to it.
> 
> Later, when at jobs, I got used to have a stash on all working branch tips.
> So I often would have several stashes in `git stash list` but only the last
> one in `gitk --all`. That annoyed me for about a year, finally I got
> working in `gitk` to show a stash status more similar to what `git stash
> list` shows.
> 
> +---+
> |The feature|
> +---+
> 
> Show all stashes in `gitk` instead of only the last one.

This seems like a good feature, although I don't use stashes myself.

> +--+
> |Why it's desirable|
> +--+
> 
> In order to have better visual control when working on repos that have
> several active branches and there are needed quick context changes between
> them with the features that `git stash [apply [STASHNAME]| list | pop
> [STASHNAME]| push | save | show [[-p] STASHNAME]]`.
> 
> In order to have a better cohesion between the mentioned features and `gitk
> --all`.
> 
> ++
> |Caveats and side effects|
> ++
> 
> With this patch several side effects happen:
> 
> * Stashes are shown even invoking `gitk`, not only when running `gitk
> --all`. If this is a problem, I can keep working in the patch to avoid this.
> 
> * The most recent stash, which was previously shown as 'stash' because its
> corresponding `git show-ref` output line reads 'refs/stash', gets shown as
> 'stash@{0}'. Not removing lines with 'stash' when calling `git show-ref`
> gets it shown both as 'stash' as usual and as 'stash@{0}'.
> 
> +--+
> |Non-obvious design choices|
> +--+
> 
> There are many improvable things consequence of never having edited
> anything Tcl before this. Besides, its working for me as a proof of
> concept, both in Debian 7 Wheezy and 8 Jessie.
> 
> The origin document of the following diff is `gitk` as it ships in Debian 8
> Jessie. I have not tried yet but if required I would be willing to rework
> it for the repo master.

A patch against the latest version in my git repo at
git://ozlabs.org/~paulus/gitk would be better.

> `gitk-stash-list-ids.sh` is a shell script that performs `git stash list
> --pretty=format:"%H"`, i.e. show rev hashes for all stashes in the fashion
> that `git rev-list --all` does its default output. I did this because I
> could not work out a better pure Tcl solution.

Hmmm, I don't want gitk to have to depend on an external script.
It should be possible to make Tcl execute the git command directly,
though (see below).

> ++
> |Unified diff follows|
> ++
> 
> 0:10:1473338052:uprego@uxio:~$ diff -u /usr/bin/gitk-deb8-vanilla
> /usr/bin/gitk-deb8-multistash
> --- /usr/bin/gitk-deb8-vanilla  2016-08-29 10:07:06.507344629 +0200
> +++ /usr/bin/gitk-deb8-multistash   2016-09-08 14:36:35.382476634 +0200
> @@ -401,6 +401,10 @@
> 
>  if {$vcanopt($view)} {
> set revs [parseviewrevs $view $vrevs($view)]
> +set stashesfd [open [concat | gitk-stash-list-ids.sh] r]

set stashesfd [open [list | git stash list {--pretty=format:%H}] r]

> +while {[gets $stashesfd stashline] >= 0} {
> +set revs [lappend revs $stashline]
> +}

Could this ever take a long time?  The UI is unresponsive while we're
in this loop.  If this is always quick (even on large repos), OK.  If
it could take a long time then we'll need a file event handler.

> if {$revs eq {}} {
> return 0
> }
> @@ -1778,7 +1782,7 @@
>  foreach v {tagids idtags headids idheads otherrefids idotherrefs} {
> catch {unset $v}
>  }
> -set refd [open [list | git show-ref -d] r]
> +set refd [open [list | git show-ref -d | grep -v stash] r]

If I had a branch called "dont-use-a-stash-for-this", would it get
filtered out by this grep?  It seems like it would, and we don't want
it to.  So the filtering needs to be more exact here.

>  while {[gets $refd line] >= 0} {
> if {[string index $line 40] ne " "} continue
> set id [string range $line 0 39]
> @@ -1811,6 +1815,16 @@
> }
>  }
>  catch 

Proposal to add a shortcut for --force-with-lease

[vi0oss]

"--force" have currently two shortcuts: "-f" and "+", hence more ergonomic.
But I expect it's better for users to use "--force-with-lease" by 
default for overriding
remote branches (e.g. cleaning up a pull request), as it rarely fails in 
normal situations.


I propose adding some shortcut for "--force-with-lease".

Options for a shortcut:

* git push origin =HEAD:master
"=" is like "+", but weaker ('+' being typed as shift+'=')
* git push -l origin HEAD:master
* git fpush origin HEAD:master

Unfortunately hijacking "-f" to mean "--force-with-lease" is not an 
option due to scripts.
"-F" is also bad choice because of it looks like "-f" with even less 
checks (like "git checkout -B" instead of "-b").


Additionally, message when doing "git push -l" or "git push -nl" may 
show the range of commits being "thrown away".

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Beneficial Proposal

[Byung W. Park]

Beneficial proposal worth US$25,500,000.00 requires mutual understanding. 
Reply to byungwoop...@163.com or byungwop...@aol.com for details. 

Byung W. Park
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proposal: allow git clone for http-hosted bundles

[Junio C Hamano]

Jeff King  writes:

> You might do better to stick a shim script in your $PATH to just
> intercept the calls to git. Hacky, but it would probably solve your
> problem with a minimal amount of code.

I recently learned about http://repo.or.cz/git.git/bundles which
is a very nicely done set of instructions ;-)
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proposal: allow git clone for http-hosted bundles

[Johannes Schindelin]

Ho mappu,

On Sun, 17 Jul 2016, mappu wrote:

> Right now it's possible to git clone a repository over http, and git
> clone a bundle from the local filesystem, but it's not possible to git
> clone a bundle hosted on http.
> 
> Would it be possible to allow this in the future? Hopefully it's only a
> minor refactor in `builtin/clone.c`.

Why don't you just give it a whirl? That would answer the question pretty
quickly...

Ciao,
Johannes
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proposal: allow git clone for http-hosted bundles

[Jeff King]

On Sun, Jul 17, 2016 at 04:41:54PM +1200, mappu wrote:

> Right now it's possible to git clone a repository over http, and git clone a
> bundle from the local filesystem, but it's not possible to git clone a
> bundle hosted on http.
> 
> Would it be possible to allow this in the future? Hopefully it's only a
> minor refactor in `builtin/clone.c`.

It's a bit more than a minor refactor. Long ago, I submitted

  http://thread.gmane.org/gmane.comp.version-control.git/185196

The tricky thing is having the http code handle the case that we get a
bundle when accessing the repository.

I think I got stalled in a "perfect is the enemy of the good" situation.
I wanted to clean up the patches to avoid spooling the bundle to disk
(because it means we temporarily required 2x disk space). But in
retrospect, it would be fine to start there, and if somebody wants to
take on resumable index-pack, that can be a separate topic.

So I never ended up getting back to it. And somehow almost 5 years have
passed yikes.

If anybody is interested in working on it, they can start from those
patches, or from the jk/bundle-fetch-wip branch at
https://github.com/peff/git. The latter has been continually rebased on
master for the past 5 years, _but_ in a fairly blind manner. I resolve
conflicts, but anything with "-wip" in the title is not part of my
regular build. So it has not been compiled nor had its tests run in all
that time. Caveat emptor.

> (Back story: I'm stuck with a git frontend that only ever calls `git clone
> ${target}` - that's Golang's `go get` - but bundles are a bit better fit for
> my request patterns than raw repositories).

You might do better to stick a shim script in your $PATH to just
intercept the calls to git. Hacky, but it would probably solve your
problem with a minimal amount of code.

-Peff
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

proposal: allow git clone for http-hosted bundles

[mappu]

Hi list,

Right now it's possible to git clone a repository over http, and git 
clone a bundle from the local filesystem, but it's not possible to git 
clone a bundle hosted on http.


Would it be possible to allow this in the future? Hopefully it's only a 
minor refactor in `builtin/clone.c`.


Regards

mappu


(Back story: I'm stuck with a git frontend that only ever calls `git 
clone ${target}` - that's Golang's `go get` - but bundles are a bit 
better fit for my request patterns than raw repositories).


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

proposal

[welcome]

 Dear Friend,
 It is obvious that we are not friend's neither are we acquaintance, hence my
 mail will be a great surprise to you, but my Intents are not to delve into
 your privacy without caution and permission,but for a project opportunity
 worth $98.6 Million that will benefit both of us mutually.If you are
 interested to know about this project, complete details of my honourable
 self and this project will be provided for your perusal.
 Email;kuntjoromich...@yahoo.com.hk

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proposal for extending smudge/clean filters with raw file access

[Joey Hess]

Junio C Hamano wrote:
> > Secondly, and harder to get around, the filename passed to the clean
> > filter is not necessarily a path to the actual existing file that is
> > being cleaned.
> 
> Either one of us is confused.  I was talking about updating the
> current "clean" implementation without changing its interface,
> i.e. gets fed via its standard input, expected to respond to its
> standard output.  There is no filename involved.

I'm talking about the %f that can be passed to the clean filter.
The context that I left out is that my clean filter could avoid reading
all of stdin, and quickly produce the cleaned result, but only if it
can examine the file that's being cleaned. Which is not currently
entirely safe to use the %f for.

There may be a way to make a clean filter that can do something useful
without reading all of stdin, and without examining the file that's
being cleaned. Maybe. Hard to see how. I don't feel such a hypothetical
clean filter is worth changing the current EPIPE behavior to support.

So I think it's better to add a separate clean-from-fs and keep the
current clean filter interface as it stands.

-- 
see shy jo


signature.asc
Description: PGP signature

Re: proposal for extending smudge/clean filters with raw file access

[Joey Hess]

Junio C Hamano wrote:
> The smudge happens to be the last to run, so it is quite true that
> it can say "Hey Git, I've written it out already".
> 
> I didn't check all codepaths to ensure that we won't need the
> smudged result in core at all, but I am guessing you did so before
> making this proposal, so in that case I would say this sounds fine.

Well, the idea is to only use smudge-to-file when the smudged content is
going to be written out to a file. Any other code paths that need to
smudge some content would use the smudge filter.

So, try_create_file would use it. Maybe some other places I have not
found yet could as well.

-- 
see shy jo


signature.asc
Description: PGP signature

Re: proposal for extending smudge/clean filters with raw file access

[Junio C Hamano]

Joey Hess  writes:

> Junio C Hamano wrote:
>> This side, I do not think we even need a new variant.  We can just
>> update the code to interact with "clean" so that it the writer to
>> the pipe ignores SIGPIPE, detects EPIPE on write(2), says "ah, the
>> other end does not need the full input to produce its output".  The
>> reader from the pipe consumes its output without failing AS LONG AS
>> the "clean" filter exits with zero (we do check its exit status,
>> right?)
>
> There are two problems with doing that. First, any clean filter that
> relied on that would not work with older versions of git.

That's a fair point.

> Secondly, and harder to get around, the filename passed to the clean
> filter is not necessarily a path to the actual existing file that is
> being cleaned.

Either one of us is confused.  I was talking about updating the
current "clean" implementation without changing its interface,
i.e. gets fed via its standard input, expected to respond to its
standard output.  There is no filename involved.

And yes, "clean-from-fs" that is spawned by us with the name of the
file in the filesystem that has the contents as its argument, must
be prepared to see a filename that does not have any relation to the
filename in the working tree.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proposal for extending smudge/clean filters with raw file access

[Joey Hess]

Junio C Hamano wrote:
> This side, I do not think we even need a new variant.  We can just
> update the code to interact with "clean" so that it the writer to
> the pipe ignores SIGPIPE, detects EPIPE on write(2), says "ah, the
> other end does not need the full input to produce its output".  The
> reader from the pipe consumes its output without failing AS LONG AS
> the "clean" filter exits with zero (we do check its exit status,
> right?)

There are two problems with doing that. First, any clean filter that
relied on that would not work with older versions of git.

Secondly, and harder to get around, the filename passed to the clean
filter is not necessarily a path to the actual existing file that is
being cleaned. For example, git hash-object --stdin --path=whatever.
So the current clean filter can't really safely rely on accessing the
file to short-circuit its cleaning. (Although it seems to mostly work..
currently..)

> We cannot do a similar "we can just unconditionally update" like I
> said on the "clean" side to "smudge", so it would need a new
> variant.  I do not want to call it anything "raw", as there is
> nothing "raw" about it.  "smudge-to-fs" would be a better name.

"raw" was just a placeholder. "clean-from-fs" and "smudge-to-fs" are
pretty descriptive.

-- 
see shy jo


signature.asc
Description: PGP signature

Re: proposal for extending smudge/clean filters with raw file access

[Junio C Hamano]

Joey Hess  writes:

> The clean filter has to consume the whole file content on stdin;
> not reading it all will make git think the clean filter failed.
> But, git-annex often doesn't need to read the whole content of a
> work-tree file in order to clean it.

This side, I do not think we even need a new variant.  We can just
update the code to interact with "clean" so that it the writer to
the pipe ignores SIGPIPE, detects EPIPE on write(2), says "ah, the
other end does not need the full input to produce its output".  The
reader from the pipe consumes its output without failing AS LONG AS
the "clean" filter exits with zero (we do check its exit status,
right?)

And I think we should do the same for any codepath that spawns
custom script and feeds it via a pipe from us (I am talking about
hooks here).  

What may require a new variant is when your clean filter may not
even need earlier contents of the file, in which case we are better
off not opening the file ourselves and slurping it into core, only
to pipe it and earlier part discarded by the filter.  "clean-from-fs"
filter that gets a path on the working tree and feeds us via pipe
would be appropriate to deal with such a requirement.

> The smudge filter has to output the whole file content to stdout.

We cannot do a similar "we can just unconditionally update" like I
said on the "clean" side to "smudge", so it would need a new
variant.  I do not want to call it anything "raw", as there is
nothing "raw" about it.  "smudge-to-fs" would be a better name.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: proposal for extending smudge/clean filters with raw file access

[Junio C Hamano]

Joey Hess <i...@joeyh.name> writes:

> I'm using smudge/clean filters in git-annex now, and it's not been an
> entirely smooth fit between the interface and what git-annex wants
> to do.
>
> The clean filter has to consume the whole file content on stdin;
> not reading it all will make git think the clean filter failed.
> But, git-annex often doesn't need to read the whole content of a
> work-tree file in order to clean it.
>
> The smudge filter has to output the whole file content to stdout. But
> git-annex often has the file's content on disk already, and could just
> move it into place in the working tree. This would save CPU and IO and
> often disk space too. But the smudge interface doesn't let git-annex use
> the efficient approach.

The smudge happens to be the last to run, so it is quite true that
it can say "Hey Git, I've written it out already".

I didn't check all codepaths to ensure that we won't need the
smudged result in core at all, but I am guessing you did so before
making this proposal, so in that case I would say this sounds fine.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

proposal for extending smudge/clean filters with raw file access

[Joey Hess]

I'm using smudge/clean filters in git-annex now, and it's not been an
entirely smooth fit between the interface and what git-annex wants
to do.

The clean filter has to consume the whole file content on stdin;
not reading it all will make git think the clean filter failed.
But, git-annex often doesn't need to read the whole content of a
work-tree file in order to clean it.

The smudge filter has to output the whole file content to stdout. But
git-annex often has the file's content on disk already, and could just
move it into place in the working tree. This would save CPU and IO and
often disk space too. But the smudge interface doesn't let git-annex use
the efficient approach.

So I propose extending the filter driver with two more optional
commands. Call them raw-clean and raw-smudge for now.

raw-clean would be like clean, but rather than being fed the whole
content of a large file on stdin, it would be passed the filename, and
can access the file itself. Like the clean filter, it outputs the
cleaned version on stdout.

raw-smudge would be like smudge, but rather than needing to output the
whole content of a large file on stdout, it would be passed a filename,
and can create that file itself.

To keep this backwards compatible, and to handle the cases where the
object being filtered is not a file on disk, the smudge and clean
filters would be required to be configured too, in order for raw-clean
and raw-smudge to be used.

It seems fairly easy to implement raw-clean. In sha1_file.c, index_path
would use raw-clean when available, while index_fd etc keep on using
the clean filter. I have not investigated what would be needed to implement
raw-smudge yet.

-- 
see shy jo
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Eric Wong]

惠轶群 <huiyi...@gmail.com> wrote:
> 2016-03-28 6:00 GMT+08:00 Eric Wong <normalper...@yhbt.net>:
> > While Gmail provides SMTP access, it was (last I was told)
> > incompatible with two-factor auth; so I've encountered users
> > unable to send patches with their normal 2FA-enabled accounts.
> 
> That's the origin of this idea of `mailto`.
> 
> In fact, you could send mail via 2FA-enabled accounts via
> "app password" metioned by Javier. But it's annoying to create
> "app password" for every client.

Since it seems possible to use 2FA with gmail, can either you
or Javier submit a patch to Documentation/git-send-email.txt
to update the Gmail-related section where
"Use gmail as the smtp server" is to describe how to use this
"app password"?

It's much easier to do than your entire GSoC proposal and would
be immediately useful to 2FA gmail users out there who don't
know this, yet or aren't running the latest git.  Thanks.





> 
> If there is a `mailto` method to send patch, you could type something
> like
> 
> git send-patch --mailto origin/master..HEAD
> 
> Then, gmail is launched with the content of patch in it. You could edit
> the list of `to` and `cc`(You could even take use of gmail contact). Then
> just send. That's all. No need to SMTP config or "app password" any
> more.
> 
> > Maybe git hackers at Google have enough pull to lobby Gmail's
> > web interface to make it easier to send plain-text patches;
> > but I would love more to see users running local mail clients
> > and even running their own SMTP servers.
> 
> Yes, this should be free with user to pick their favorite mail client.
> 
> >> That may not be a "Git" project, but GSoC is not limited to Git ;-)
> >
> > Completely agreed; email is critical to decentralized development;
> > but I don't believe decentralization is in the best interests of
> > any large and powerful corporation.
> >
> > IMHO, what we need is a SoIS: Summer of Independent Sysadmins :>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Johannes Schindelin]

Hi,

On Wed, 30 Mar 2016, Ævar Arnfjörð Bjarmason wrote:

> Correct me if I'm wrong but don't we basically have 4 kinds of users
> using git-send-email:
> 
> 1) Those who get it from a binary Windows package (is it even packaged there?)

It is. And reportedly working fine. But in the pre-MSYS2 times it was a
major pain in the butt due to the dependencies.

Ciao,
Dscho

Re: [GSoC] A late proposal: a modern send-email

[Ævar Arnfjörð Bjarmason]

On Tue, Mar 29, 2016 at 6:17 AM, 惠轶群 <huiyi...@gmail.com> wrote:
> 2016-03-29 0:49 GMT+08:00 Ævar Arnfjörð Bjarmason <ava...@gmail.com>:
>> On Sat, Mar 26, 2016 at 3:13 AM, 惠轶群 <huiyi...@gmail.com> wrote:
>>> 2016-03-26 2:16 GMT+08:00 Junio C Hamano <gits...@pobox.com>:
>>>> 惠轶群 <huiyi...@gmail.com> writes:
>>>>
>>>>> # Purpose
>>>>> The current implementation of send-email is based on perl and has only
>>>>> a tui, it has two problems:
>>>>> - user must install a ton of dependencies before submit a single patch.
>>>>> - tui and parameter are both not quite friendly to new users.
>>>>
>>>> Is "a ton of dependencies" true?  "apt-cache show git-email"
>>>> suggests otherwise.  Is "a ton of dependencies" truly a problem?
>>>> "apt-get install" would resolve the dependencies for you.
>>>
>>> There are three perl packages needed to send patch through gmail:
>>> - perl-mime-tools
>>> - perl-net-smtp-ssl
>>> - perl-authen-sasl
>>>
>>> Yes, not too many, but is it better none of them?
>>>
>>> What's more, when I try to send mails, I was first disrupted by
>>> "no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
>>> Then I think, why not just a mailto link?
>>
>> I think your proposal should clarify a bit who these users are that
>> find it too difficult to install these perl module dependencies. Users
>> on OSX & Windows I would assume, because in the case of Linux distros
>> getting these is the equivalent of an apt-get command away.
>
> In fact, I'm not familiar with the build for OSX or Windows.

The core of your proposal rests on the assumption that
git-send-email's implementation is problematic because it has a "ton
of dependencies", and that this must be dealt with by implementing an
alternate E-Mail transport method.

But you don't go into how this is a practical issue for users exactly,
which is the rest of the proposal. I.e. "make it friendly for users".
Let's leave the question of creating an E-Mail GUI that's shipped with
Git aside.

Correct me if I'm wrong but don't we basically have 4 kinds of users
using git-send-email:

1) Those who get it from a binary Windows package (is it even packaged there?)
2) Also a binary package, but for for OSX
3) Users installing it via their Linux distribution's package system
4) Users building it from source on Windows/OSX/Linux.

I'm in group #3 myself for the purposes of using git-send-email and
have never had issues with its dependencies because my distro's
package management takes care of it for me.

I don't know what the status is of packaging it is on #1 and #2, but
that's what I'm asking about in my question, if this becomes a
non-issue for those two groups (if it isn't already) isn't this
question of dependencies a non-issue?

I.e. why does it matter if git-send-email has N dependencies if those
N are either packaged with the common Windows/OSX packages that most
users use, or installed as dependencies by their *nix distro?

 Group #4 is small enough and likely to be a git.git contributor or
distro package maintainer anyway that this issue doesn't matter for
them.

>> If installing these dependencies is hard for users perhaps a better
>> thing to focus on is altering the binary builds on Git for platforms
>> that don't have package systems to include these dependencies.
>
> Why `mailto` not a good choice? I'm confusing.

I'm not saying having this mailto: method you're proposing isn't good
in itself, I think it would be very useful to be able to magically
open git-send-email output in your favorite E-Mail client for editing
before sending it off like you usually send E-Mail.

Although I must say I'd be seriously surprised if the likes of git
formatted patches survive contact with popular E-Mail clients when the
body is specified via the body=* parameter, given that we're sending
pretty precisely formatted content and most mailers are very eager to
wrap lines or otherwise munge input.

I'm mainly trying to get to the bottom of this dependency issue you're
trying to solve.

>> In this case it would mean shipping a statically linked OpenSSL since
>> that's what these perl SSL packages eventually depend on.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[惠轶群]

2016-03-29 0:49 GMT+08:00 Ævar Arnfjörð Bjarmason <ava...@gmail.com>:
> On Sat, Mar 26, 2016 at 3:13 AM, 惠轶群 <huiyi...@gmail.com> wrote:
>> 2016-03-26 2:16 GMT+08:00 Junio C Hamano <gits...@pobox.com>:
>>> 惠轶群 <huiyi...@gmail.com> writes:
>>>
>>>> # Purpose
>>>> The current implementation of send-email is based on perl and has only
>>>> a tui, it has two problems:
>>>> - user must install a ton of dependencies before submit a single patch.
>>>> - tui and parameter are both not quite friendly to new users.
>>>
>>> Is "a ton of dependencies" true?  "apt-cache show git-email"
>>> suggests otherwise.  Is "a ton of dependencies" truly a problem?
>>> "apt-get install" would resolve the dependencies for you.
>>
>> There are three perl packages needed to send patch through gmail:
>> - perl-mime-tools
>> - perl-net-smtp-ssl
>> - perl-authen-sasl
>>
>> Yes, not too many, but is it better none of them?
>>
>> What's more, when I try to send mails, I was first disrupted by
>> "no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
>> Then I think, why not just a mailto link?
>
> I think your proposal should clarify a bit who these users are that
> find it too difficult to install these perl module dependencies. Users
> on OSX & Windows I would assume, because in the case of Linux distros
> getting these is the equivalent of an apt-get command away.

In fact, I'm not familiar with the build for OSX or Windows.

> If installing these dependencies is hard for users perhaps a better
> thing to focus on is altering the binary builds on Git for platforms
> that don't have package systems to include these dependencies.

Why `mailto` not a good choice? I'm confusing.

> In this case it would mean shipping a statically linked OpenSSL since
> that's what these perl SSL packages eventually depend on.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Ævar Arnfjörð Bjarmason]

On Sat, Mar 26, 2016 at 3:13 AM, 惠轶群 <huiyi...@gmail.com> wrote:
> 2016-03-26 2:16 GMT+08:00 Junio C Hamano <gits...@pobox.com>:
>> 惠轶群 <huiyi...@gmail.com> writes:
>>
>>> # Purpose
>>> The current implementation of send-email is based on perl and has only
>>> a tui, it has two problems:
>>> - user must install a ton of dependencies before submit a single patch.
>>> - tui and parameter are both not quite friendly to new users.
>>
>> Is "a ton of dependencies" true?  "apt-cache show git-email"
>> suggests otherwise.  Is "a ton of dependencies" truly a problem?
>> "apt-get install" would resolve the dependencies for you.
>
> There are three perl packages needed to send patch through gmail:
> - perl-mime-tools
> - perl-net-smtp-ssl
> - perl-authen-sasl
>
> Yes, not too many, but is it better none of them?
>
> What's more, when I try to send mails, I was first disrupted by
> "no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
> Then I think, why not just a mailto link?

I think your proposal should clarify a bit who these users are that
find it too difficult to install these perl module dependencies. Users
on OSX & Windows I would assume, because in the case of Linux distros
getting these is the equivalent of an apt-get command away.

If installing these dependencies is hard for users perhaps a better
thing to focus on is altering the binary builds on Git for platforms
that don't have package systems to include these dependencies.

In this case it would mean shipping a statically linked OpenSSL since
that's what these perl SSL packages eventually depend on.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[惠轶群]

2016-03-28 6:00 GMT+08:00 Eric Wong :
> Junio C Hamano  wrote:
>> 惠轶群  writes:
>> > - Build a simple email client (maybe a web components based web app or
>> > wxwidgets based GUI client, they are both cross-platform) which is
>> > easy to use for sending patch without disrupting the mailbox format.
>>
>> I suspect it would yield a better result if the plan were to update
>> a popular email client and make it possible to tell it to read an
>> existing text file (i.e. mbox) without corrupting its contents.
>> People do not have to learn a new mail client if done that way.
>
> Another bigger problem is people rely heavily on webmail UIs
> nowadays instead of running any local mail clients :<

I know many heavily email user choose to migrate to client such as
alpine or mutt after using gmail for a long time.

> While Gmail provides SMTP access, it was (last I was told)
> incompatible with two-factor auth; so I've encountered users
> unable to send patches with their normal 2FA-enabled accounts.

That's the origin of this idea of `mailto`.

In fact, you could send mail via 2FA-enabled accounts via
"app password" metioned by Javier. But it's annoying to create
"app password" for every client.

If there is a `mailto` method to send patch, you could type something
like

git send-patch --mailto origin/master..HEAD

Then, gmail is launched with the content of patch in it. You could edit
the list of `to` and `cc`(You could even take use of gmail contact). Then
just send. That's all. No need to SMTP config or "app password" any
more.

> Maybe git hackers at Google have enough pull to lobby Gmail's
> web interface to make it easier to send plain-text patches;
> but I would love more to see users running local mail clients
> and even running their own SMTP servers.

Yes, this should be free with user to pick their favorite mail client.

>> That may not be a "Git" project, but GSoC is not limited to Git ;-)
>
> Completely agreed; email is critical to decentralized development;
> but I don't believe decentralization is in the best interests of
> any large and powerful corporation.
>
> IMHO, what we need is a SoIS: Summer of Independent Sysadmins :>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Javier Domingo Cansino]

> While Gmail provides SMTP access, it was (last I was told)
> incompatible with two-factor auth; so I've encountered users
> unable to send patches with their normal 2FA-enabled accounts.
>
> Maybe git hackers at Google have enough pull to lobby Gmail's
> web interface to make it easier to send plain-text patches;
> but I would love more to see users running local mail clients
> and even running their own SMTP servers.

Just in case, I use git send email with my gmail account and I have 2
step auth configured. The only "limitation" however is that you have
to create a password for each email client on it's own. If you own a
gmail account, go to
https://security.google.com/settings/security/apppasswords to create a
password for the email client.

-- 
Javier Domingo Cansino
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Eric Wong]

Junio C Hamano  wrote:
> 惠轶群  writes:
> > - Build a simple email client (maybe a web components based web app or
> > wxwidgets based GUI client, they are both cross-platform) which is
> > easy to use for sending patch without disrupting the mailbox format.
> 
> I suspect it would yield a better result if the plan were to update
> a popular email client and make it possible to tell it to read an
> existing text file (i.e. mbox) without corrupting its contents.
> People do not have to learn a new mail client if done that way.

Another bigger problem is people rely heavily on webmail UIs
nowadays instead of running any local mail clients :<

While Gmail provides SMTP access, it was (last I was told)
incompatible with two-factor auth; so I've encountered users
unable to send patches with their normal 2FA-enabled accounts.

Maybe git hackers at Google have enough pull to lobby Gmail's
web interface to make it easier to send plain-text patches;
but I would love more to see users running local mail clients
and even running their own SMTP servers.

> That may not be a "Git" project, but GSoC is not limited to Git ;-)

Completely agreed; email is critical to decentralized development;
but I don't believe decentralization is in the best interests of
any large and powerful corporation.

IMHO, what we need is a SoIS: Summer of Independent Sysadmins :>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[惠轶群]

2016-03-26 17:52 GMT+08:00 惠轶群 :
> 2016-03-26 14:18 GMT+08:00 Pranit Bauva :
>> On Sat, Mar 26, 2016 at 7:43 AM, 惠轶群  wrote:
>>> 2016-03-26 2:16 GMT+08:00 Junio C Hamano :
 惠轶群  writes:

> # Purpose
> The current implementation of send-email is based on perl and has only
> a tui, it has two problems:
> - user must install a ton of dependencies before submit a single patch.
> - tui and parameter are both not quite friendly to new users.

 Is "a ton of dependencies" true?  "apt-cache show git-email"
 suggests otherwise.  Is "a ton of dependencies" truly a problem?
 "apt-get install" would resolve the dependencies for you.
>>>
>>> There are three perl packages needed to send patch through gmail:
>>> - perl-mime-tools
>>> - perl-net-smtp-ssl
>>> - perl-authen-sasl
>>>
>>> Yes, not too many, but is it better none of them?
>>
>> Are you sure using a GUI does not have any dependencies?
>>
>>> What's more, when I try to send mails, I was first disrupted by
>>> "no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
>>> Then I think, why not just a mailto link?
>>>
> # Plan
> So I propose to implement following:
> - Allow user to send mail via a [`mailto`
> link](https://en.wikipedia.org/wiki/Mailto). so that users could
> complete the mail in their favorite email clients such as gmail, mutt,
> alpine and even gmail for android through

 IIRC, GMail on Android is incapable of sending a "text/plain", so
 that part may not fly well.
>>>
>>> Really? As much as I known, GMail on Android is capable of sending
>>> a "text/plain" while Inbox is not.
>>
>> How do you plan in integrating GMail on Android so that it can send
>> patches which exists on your computer?
>
> No, if you could have termux a try, you will find that it's suitable for 
> simple
> development. it has a apt, so you could have clang, neovim, tmux, cmake
> and so on.
>
> In fact, I recently use my nexus 7 with termux as a portable
> development environment.
> A bluetooth keyboard is needed, of course.

This is not applicable to all people, but It make the git more free, isn't it?

> - Build a simple email client (maybe a web components based web app or
> wxwidgets based GUI client, they are both cross-platform) which is
> easy to use for sending patch without disrupting the mailbox format.
>>
>> I think introducing a GUI may lead to much more dependencies. Many git
>> developers already have perl packages in their system but they don't
>> have wxwidgets.
>
> wxwidgets seems not a good choice. But if I build the GUI via web app,
> I could import required js and css from Internet directly, so the users do
> not need the dependencies on their computer.
>
 I suspect it would yield a better result if the plan were to update
 a popular email client and make it possible to tell it to read an
 existing text file (i.e. mbox) without corrupting its contents.
 People do not have to learn a new mail client if done that way.
>>>
>>> Maybe a plugin? I'm not sure.
>>
>> You could make a plugin. That would simply things.
>>
>>> If above `mail-to` is implemented, user could just using any mail
>>> client, but a mail client adaptive for patch would be better:
>>> - Do not allow user to edit the diff part
>>> - always 'plan/text'
>>> - visual
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[惠轶群]

To make it clearer:

1. there are so many great mail clients, why not let these experts to handle the
dispatch of mail?

For example, as a One-time-password user of gmail, I have to generate an app
password for each terminal device I'd like to send patch on.

It's more important that I could not use my favorite mail client which
has contact
list, familiar interface and more personal configuration.

Why not let one program just do one thing and do it best?

In my opinion, `mailto` is a suitable method. git is one of the best
version control
systems. So it forms basic structure of the mail according to the
patches and passes it
to the best mail client. How amazing.

2. But this workload is not enough for whole summer. I thought I could do other
interesting things based on the `mailto` link.

mail client is a straightforward choice.

In fact, I meant to implement an interface more friendly to new users.
no matter how
it is implemented or what it is based on. The first time I send patch
via gmail is not a
happy memory.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[惠轶群]

2016-03-26 14:18 GMT+08:00 Pranit Bauva :
> On Sat, Mar 26, 2016 at 7:43 AM, 惠轶群  wrote:
>> 2016-03-26 2:16 GMT+08:00 Junio C Hamano :
>>> 惠轶群  writes:
>>>
 # Purpose
 The current implementation of send-email is based on perl and has only
 a tui, it has two problems:
 - user must install a ton of dependencies before submit a single patch.
 - tui and parameter are both not quite friendly to new users.
>>>
>>> Is "a ton of dependencies" true?  "apt-cache show git-email"
>>> suggests otherwise.  Is "a ton of dependencies" truly a problem?
>>> "apt-get install" would resolve the dependencies for you.
>>
>> There are three perl packages needed to send patch through gmail:
>> - perl-mime-tools
>> - perl-net-smtp-ssl
>> - perl-authen-sasl
>>
>> Yes, not too many, but is it better none of them?
>
> Are you sure using a GUI does not have any dependencies?
>
>> What's more, when I try to send mails, I was first disrupted by
>> "no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
>> Then I think, why not just a mailto link?
>>
 # Plan
 So I propose to implement following:
 - Allow user to send mail via a [`mailto`
 link](https://en.wikipedia.org/wiki/Mailto). so that users could
 complete the mail in their favorite email clients such as gmail, mutt,
 alpine and even gmail for android through
>>>
>>> IIRC, GMail on Android is incapable of sending a "text/plain", so
>>> that part may not fly well.
>>
>> Really? As much as I known, GMail on Android is capable of sending
>> a "text/plain" while Inbox is not.
>
> How do you plan in integrating GMail on Android so that it can send
> patches which exists on your computer?

No, if you could have termux a try, you will find that it's suitable for simple
development. it has a apt, so you could have clang, neovim, tmux, cmake
and so on.

In fact, I recently use my nexus 7 with termux as a portable
development environment.
A bluetooth keyboard is needed, of course.

 - Build a simple email client (maybe a web components based web app or
 wxwidgets based GUI client, they are both cross-platform) which is
 easy to use for sending patch without disrupting the mailbox format.
>
> I think introducing a GUI may lead to much more dependencies. Many git
> developers already have perl packages in their system but they don't
> have wxwidgets.

wxwidgets seems not a good choice. But if I build the GUI via web app,
I could import required js and css from Internet directly, so the users do
not need the dependencies on their computer.

>>> I suspect it would yield a better result if the plan were to update
>>> a popular email client and make it possible to tell it to read an
>>> existing text file (i.e. mbox) without corrupting its contents.
>>> People do not have to learn a new mail client if done that way.
>>
>> Maybe a plugin? I'm not sure.
>
> You could make a plugin. That would simply things.
>
>> If above `mail-to` is implemented, user could just using any mail
>> client, but a mail client adaptive for patch would be better:
>> - Do not allow user to edit the diff part
>> - always 'plan/text'
>> - visual
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Pranit Bauva]

On Sat, Mar 26, 2016 at 7:43 AM, 惠轶群  wrote:
> 2016-03-26 2:16 GMT+08:00 Junio C Hamano :
>> 惠轶群  writes:
>>
>>> # Purpose
>>> The current implementation of send-email is based on perl and has only
>>> a tui, it has two problems:
>>> - user must install a ton of dependencies before submit a single patch.
>>> - tui and parameter are both not quite friendly to new users.
>>
>> Is "a ton of dependencies" true?  "apt-cache show git-email"
>> suggests otherwise.  Is "a ton of dependencies" truly a problem?
>> "apt-get install" would resolve the dependencies for you.
>
> There are three perl packages needed to send patch through gmail:
> - perl-mime-tools
> - perl-net-smtp-ssl
> - perl-authen-sasl
>
> Yes, not too many, but is it better none of them?

Are you sure using a GUI does not have any dependencies?

> What's more, when I try to send mails, I was first disrupted by
> "no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
> Then I think, why not just a mailto link?
>
>>> # Plan
>>> So I propose to implement following:
>>> - Allow user to send mail via a [`mailto`
>>> link](https://en.wikipedia.org/wiki/Mailto). so that users could
>>> complete the mail in their favorite email clients such as gmail, mutt,
>>> alpine and even gmail for android through
>>
>> IIRC, GMail on Android is incapable of sending a "text/plain", so
>> that part may not fly well.
>
> Really? As much as I known, GMail on Android is capable of sending
> a "text/plain" while Inbox is not.

How do you plan in integrating GMail on Android so that it can send
patches which exists on your computer?

>>> - Build a simple email client (maybe a web components based web app or
>>> wxwidgets based GUI client, they are both cross-platform) which is
>>> easy to use for sending patch without disrupting the mailbox format.

I think introducing a GUI may lead to much more dependencies. Many git
developers already have perl packages in their system but they don't
have wxwidgets.

>> I suspect it would yield a better result if the plan were to update
>> a popular email client and make it possible to tell it to read an
>> existing text file (i.e. mbox) without corrupting its contents.
>> People do not have to learn a new mail client if done that way.
>
> Maybe a plugin? I'm not sure.

You could make a plugin. That would simply things.

> If above `mail-to` is implemented, user could just using any mail
> client, but a mail client adaptive for patch would be better:
> - Do not allow user to edit the diff part
> - always 'plan/text'
> - visual
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] Proposal

[惠轶群]

I had updated the proposal before deadline, if someone is interesting.

2016-03-25 15:12 GMT+08:00 惠轶群 <huiyi...@gmail.com>:
> Well, I should have done some search before ask.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[惠轶群]

2016-03-26 2:16 GMT+08:00 Junio C Hamano :
> 惠轶群  writes:
>
>> # Purpose
>> The current implementation of send-email is based on perl and has only
>> a tui, it has two problems:
>> - user must install a ton of dependencies before submit a single patch.
>> - tui and parameter are both not quite friendly to new users.
>
> Is "a ton of dependencies" true?  "apt-cache show git-email"
> suggests otherwise.  Is "a ton of dependencies" truly a problem?
> "apt-get install" would resolve the dependencies for you.

There are three perl packages needed to send patch through gmail:
- perl-mime-tools
- perl-net-smtp-ssl
- perl-authen-sasl

Yes, not too many, but is it better none of them?

What's more, when I try to send mails, I was first disrupted by
"no perl-mime-tools" then by "no perl-net-smtp-ssl or perl-authen-sasl".
Then I think, why not just a mailto link?

>> # Plan
>> So I propose to implement following:
>> - Allow user to send mail via a [`mailto`
>> link](https://en.wikipedia.org/wiki/Mailto). so that users could
>> complete the mail in their favorite email clients such as gmail, mutt,
>> alpine and even gmail for android through
>
> IIRC, GMail on Android is incapable of sending a "text/plain", so
> that part may not fly well.

Really? As much as I known, GMail on Android is capable of sending
a "text/plain" while Inbox is not.

>> - Build a simple email client (maybe a web components based web app or
>> wxwidgets based GUI client, they are both cross-platform) which is
>> easy to use for sending patch without disrupting the mailbox format.
>
> I suspect it would yield a better result if the plan were to update
> a popular email client and make it possible to tell it to read an
> existing text file (i.e. mbox) without corrupting its contents.
> People do not have to learn a new mail client if done that way.

Maybe a plugin? I'm not sure.

If above `mail-to` is implemented, user could just using any mail
client, but a mail client adaptive for patch would be better:
- Do not allow user to edit the diff part
- always 'plan/text'
- visual

> That may not be a "Git" project, but GSoC is not limited to Git ;-)

Sorry, but I do not really mean to implement a general mail client.
In fact, this is a GUI front end for git send-mail and may be placed
under contrib/
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GSoC] A late proposal: a modern send-email

[Junio C Hamano]

惠轶群  writes:

> # Purpose
> The current implementation of send-email is based on perl and has only
> a tui, it has two problems:
> - user must install a ton of dependencies before submit a single patch.
> - tui and parameter are both not quite friendly to new users.

Is "a ton of dependencies" true?  "apt-cache show git-email"
suggests otherwise.  Is "a ton of dependencies" truly a problem?
"apt-get install" would resolve the dependencies for you.

> # Plan
> So I propose to implement following:
> - Allow user to send mail via a [`mailto`
> link](https://en.wikipedia.org/wiki/Mailto). so that users could
> complete the mail in their favorite email clients such as gmail, mutt,
> alpine and even gmail for android through

IIRC, GMail on Android is incapable of sending a "text/plain", so
that part may not fly well.

> - Build a simple email client (maybe a web components based web app or
> wxwidgets based GUI client, they are both cross-platform) which is
> easy to use for sending patch without disrupting the mailbox format.

I suspect it would yield a better result if the plan were to update
a popular email client and make it possible to tell it to read an
existing text file (i.e. mbox) without corrupting its contents.
People do not have to learn a new mail client if done that way.

That may not be a "Git" project, but GSoC is not limited to Git ;-)
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[GSoC] A late proposal: a modern send-email

[惠轶群]

I know it's tooo late to post proposal here, but I'd like to let my
idea known by
others.

# Summary
Although many projects hosted on github or gitlab adopt the
fork-and-merge working flow, email is still  irreplaceable for its
compatibility. However, the send-email module is old and difficult for
new users. I’d like to improve it and implement a more modern
interface for git-send-email.

# Purpose
The current implementation of send-email is based on perl and has only
a tui, it has two problems:
- user must install a ton of dependencies before submit a single patch.
- tui and parameter are both not quite friendly to new users.

# Plan
So I propose to implement following:
- Allow user to send mail via a [`mailto`
link](https://en.wikipedia.org/wiki/Mailto). so that users could
complete the mail in their favorite email clients such as gmail, mutt,
alpine and even gmail for android through
[termux](https://termux.com/) (termux is a terminal emulator on
Android and has an API to call android system to open a link).
- Build a simple email client (maybe a web components based web app or
wxwidgets based GUI client, they are both cross-platform) which is
easy to use for sending patch without disrupting the mailbox format.

Here is my proposal:

https://docs.google.com/document/d/1PbaKJOUWjpAh0JEunqhKWiWW4SAr_-7DH_ss4WkgDGs/edit
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: GSoC 2016 | Proposal | Incremental Rewrite of git bisect

[Pranit Bauva]

On Fri, Mar 25, 2016 at 5:10 PM, Christian Couder
<christian.cou...@gmail.com> wrote:
> On Fri, Mar 25, 2016 at 11:15 AM, Pranit Bauva <pranit.ba...@gmail.com> wrote:
>>> - you will add an option to "git bisect--helper" to perform what the
>>> git-bisect.sh function did, and
>>> - you will create a test script for "git bisect--helper" in which you
>>> will test each option?
>>
>> I had very initially planned to do this. But Matthieu pointed out that
>> it would be much better to use the existing test suite rather than
>> creating one which can lead to less coverage.
>
> Ok, then perhaps:
>
> - you will add tests to existing test scripts, so that each "git
> bisect--helper" option is (indirectly) tested.
Yes. I will mention this in the proposal also. Thanks for reminding.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html