Re: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad

[Mads Kiilerich]

Thank you.

I pushed that to the stable branch.

/Mads


On 03/09/2024 11:29, Valentin Kleibel wrote:

Hi,

I propose handling it like 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/8b7c72355750fcf791cd4abfc332e26e556aee90 
.


We don't really *know* if the error came from the URL encoding, so I 
prefer to just catch the error we actually saw, and state the fact 
that we got a decode exception while processing request parameters.


Do you agree patch that would be fine - and that the attribution is 
fine?


Thanks for your work, that patch looks/works good and the attribution 
is fine with me.


I'm not positively sure webob can't choke on decoding anything else. 
That could perhaps be nice to test/review too. But that is apparently 
not a real problem (yet).


I'm pretty sure it can, but one might need to send a wrong encoding in 
the Content-type header.
For the URL-parameters, in contrast, every byte sequence encoded will 
always be interpreted as UTF-8, even before csrf or auth checks.


Valentin



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad

[Mads Kiilerich]

Hi

I propose handling it like 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/8b7c72355750fcf791cd4abfc332e26e556aee90 
.


We don't really *know* if the error came from the URL encoding, so I 
prefer to just catch the error we actually saw, and state the fact that 
we got a decode exception while processing request parameters.


Do you agree patch that would be fine - and that the attribution is fine?

I'm not positively sure webob can't choke on decoding anything else. 
That could perhaps be nice to test/review too. But that is apparently 
not a real problem (yet).


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] fix: UnicodeDecodeError: can't decode byte 0xad

[Mads Kiilerich]

Hi

Thanks for the report and the patch.

We could also catch this exception in the big try-except clause in 
__call__, and we could catch the more generic UnicodeError. But that 
would perhaps catch too much - also things that really are programming 
errors and shouldn't give a 400 reply.


I think I would prefer to just catch this Unicode error if it happens, 
rather than trying to trigger it early. Perhaps by wrapping the call of 
_basic_security_checks. Do  you think that would catch too much or too 
little?


/Mads



On 26/08/2024 17:06, Valentin Kleibel wrote:

Hi,

we have recently noticed a lot of errors in Kallithea from probing for 
a php vulnerability [1] looking like:
"WebApp Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 
0xad in position 0: invalid start byte"


This can be reproduced with curl:
curl https://example.com/?%AD

The error stems from webob naively trying to utf-8 decode all 
%-encoded bytes in URL-parameters.


In my opinion this exception should be handled and a error 400 should 
be returned.


Attached you can find a small patch i created to check for this in 
kallithea/controllers/base.py:_basic_security_checks().


Best Regards,
Valentin


[1] 
https://devco.re/blog/2024/06/06/security-alert-cve-2024-4577-php-cgi-argument-injection-vulnerability-en/


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Authorization required for cloning Kallithea with Mercurial >= 6.5.0

[Mads Kiilerich]

Hi Branko

Thanks for reporting.

I have indeed recently caught and pushed support for latest Mercurial 
version and updated the server. But it was apparently not complete.


This should be fixed by 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/0245e0ebddd09c6bcc7b1cd74708cf19a49ee57e 
- can you confirm that?


Regards,
Mads


On 23/07/2024 13:17, Branko Majic wrote:

Hello,

Attempting to clone the official Kallithea repository (at
https://kallithea-scm.org/repos/kallithea) anonymously with newer
versions of Mercurial seems to fail, and prompts user for credentials:

%
(hgtest) branko@che:~/projects/hgtest$ hg version
Mercurial Distributed SCM (version 6.5.3)
(see https://mercurial-scm.org for more information)

Copyright (C) 2005-2023 Olivia Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Enabled extensions:

   histedit  internal
   strip internal
(hgtest) branko@che:~/projects/hgtest$ hg clone 
https://kallithea-scm.org/repos/kallithea
destination directory: kallithea
http authorization required for https://kallithea-scm.org/repos/kallithea
realm: Kallithea authentication
user:
%

Based on the realm part - I assume this could be something related to
Kerberos?

 From what I can tell, this happens only with "newer" versions of
Mercurial - starting with version 6.5.0. I've tested with 6.4.5 and did not get
prompted for credentials. This testing was all done from within dedicated
Python virtual environment.

Since I am not experiencing the same issue with my test installation of
Kallithea, I assume this is related to how the official Kallithea
instance has been configured. Maybe if it detects that client can do
Kerberos-based authentication it insists on using it?

Best regards,
Branko

P.S.
Of course, could be it's not Kerberos at all, just me jumping to
conclusions. :)


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: error while pulling changes from remote mercurial repo

[Mads Kiilerich]

On 21/02/2024 10:41, Ed Wong wrote:

File "/var/www/kali/venv/lib64/python3.9/site-packages/mercurial/ui.py",
line 1299, in _writenobuf
   dest.write(msg)
File
"/var/www/kali/venv/lib64/python3.9/site-packages/mercurial/utils/procutil.py",
line 115, in write
   c = write1(m[total_written:])
peError: write() argument 1 must be read-only bytes-like object, not
memoryview



That exception is quite deep inside Mercurial, in a version that is 
quite old now. But also, it could perhaps be something with the WSGI 
stack in your setup. It seems like you are doing something is quite 
common, but the problem is not common. Something special must be 
triggering it. But also, the last official release is indeed quite old. 
If the problem can be reproduced and hasn't been fixed on the stable 
branch, we can fix it there.




Thinking that this might have been some compatibility issue between
the old Kallithea managed repos and the 0.7.0, I thought I'd create
a new repository.



In my opinion, that seems unlikely to solve the problem.



File
"/var/www/kali/venv/lib64/python3.9/site-packages/celery/app/base.py",
line 776, in send_task
  """
  TypeError: as_task_v2() got an unexpected keyword argument 'ignore_result'


At this point, I'm kinda stuck.   I have Celery == 5.0.5.



That could seem like https://github.com/celery/celery/issues/6829 - are 
you sure you are using celery 5.0.5 ?



On 25/02/2024 14:32, Branko Majic wrote:

So... I don't know what the exact issue is you are facing here, but
based on some personal experiences The released version of
Kallithea (0.7.0) is quite outdated from the code perspective when
compared to what is currently in the repository tip.



Yes - the stable branch is very stable ... while also supporting more 
recent and bugfixed Mercurial versions, so I would recommend using that 
instead of the aging latest official release. 
https://kallithea.readthedocs.io/en/stable/installation.html#installation-from-repository-source


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: 500 Internal Server Error with 2-way diff and commit that renames file to its parent directory

[Mads Kiilerich]

Thanks.

Without digging deep at all, a quick answer:

Evidently, that is a corner case that doesn't have any/sufficient test 
coverage. But it might not be worth it to add it.


First of all, it is a bug that this NodeError shows up as a server 
error. It should show a nice error, as it (hopefully) happens if you 
specify an invalid repo, and invalid revision, or a completely invalid 
file path. Even a 404 would be better, but Kallithea rarely does that on 
invalid input data. Perhaps the exception handling is missing for 
NodeError or in general. Perhaps it should handle NodeError too, or 
perhaps it should raise another exception.


It is also a bug that the message in the exception isn't suitable for 
display to the user. That should be fixed too.


In general, when showing the diff of a file that has been added or 
removed, it would be nice to clearly show it as added / removed and a 
diff against an empty file. That seems to be approximately what happens. 
But perhaps not making it sufficiently clear if a file was added/removed 
or just is empty. There might be something there that would be nice to 
get fixed too.


If showing the diff of a file that either has been moved to or from the 
file name (or copied), it would be nice to show the diff across the 
rename and make it clear that it has been renamed. At least in URLs that 
can be reached from the Kallithea UI. I'm not sure that is handled 
correctly in all cases. Perhaps not at all. We should perhaps spend more 
time looking at the changeset data before deciding exactly which diff to 
show. But also, while it might be feasible for Mercurial where renames 
are tracked and easily available, it might not be feasible for Git where 
renames have to be guessed from looking at the changesets. This might be 
nice-to-have area, but perhaps not worth it.


For both add/remove and rename to/from, I don't think Kallithea should 
behave differently if there is a directory instead of "nothing". We 
don't even have to tell about the directory. Both Mercurial and Git only 
track files, and directories only exist as collections of files. 
Perhaps, if the found node is a dir, just raise NodeDoesNotExistError 
and go to the "file not found" branch?


Also, there is no prior art for comparing directories in Kallithea. 
Introducing that would not be a bug fix but proper development. I don't 
think that is worth it. Just throw a clear error message.


Great if you can work on fixing this area. Please try to separate the 
fixes for each issue in clean commits.


/Mads




On 12/11/2023 00:40, Branko Majic wrote:

Ok, I've managed to hack up a quick fix for this, but... It does look a
bit ugly. Feels like there's too much repetitive code in there as well.
The patch is attached to the mail.

Looking at what the server does when you pass in a path that is
directory in both changesets, it really should not throw an internal
error for it. It should probably just do a diff on two changesets that
are basically empty (like the whole NodeDoesNotExistError handling), or
try to display some form of information to user (or maybe give a 404 or
something similar).

So, before I start delving into this with proper patch, any opinions on
it? :)

Best regards,
Branko


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Website down?

[Mads Kiilerich]

On 11/11/2023 12:25, Branko Majic wrote:

Since a couple of days ago, I don't seem to be able to access the main
Kallithea website (https://kallithea-scm.org/). The connections keeps
timing out.

And it seems to be back now... I guess somebody restarted a service or
two or...? :)



Yeah - sorry about that. I'm not sure what happened. It is a small 
overloaded server. I will keep an eye on it.


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: mercurial cloning from git bash

[Mads Kiilerich]

Hi

It seems odd that the problem should be caused by the upgrade.

Celery is not involved when cloning, so that doesn't make any difference.

If the problem only was seen with http, then I would blame the proxy. 
Both the problems with ssh rules that out.


Especially for ssh, Kallithea gets out of the way once the connection 
has been established and authenticated. It thus seems like the problem 
is somewhere in the general system. Somewhere that can impact both 
connections to the http stack and to the ssh server.


My best guess would be some kind of network problem, perhaps a timeout 
in a firewall or NAT router. (As the error message suggest.)


As a data point, you could try making the same clone commands from 
another account on the server machine or a machine next to it. (But it 
will probably also be faster that way, so if it works, you can't know if 
you bypassed the location of the problem, or if you just were fast 
enough to avoid hitting the problem...)


If you have a real ssh account on the server machine and can read the 
repos from it, then you can also try to clone from 
ssh://username@IP//srv/repos/reponame without using Kallithea at all.


You can also try with another more recent Mercurial version on the 
client side, perhaps on another machine.


/Mads



On 19/07/2023 11:01, Ed Wong wrote:

Hi,

I finally was able to upgrade an old Kallithea setup to the
latest version and migrated the old repositories to the
new system.  That said, I haven't setup Celery as
that requires either RabbitMQ or redis.

However, I'm having some difficulties in cloning off
a mercurial repository (based on Mozilla's mozilla-central).

I tried the following:

1)
hg clone --debug ssh:/// newrepo

  At this point, it would spew out a long list of
entries such as "add changeset "; but would
hang at some changeset and would just display
"ad"

2) hg clone --debug http:/// newrepo

   Does the same thing as #1, but at one point it
dumps the following:

.
.
add changeset 5f2a7ad96962
bundle2-input-part: total payload size 85113701
transaction abort!
rollback completed
(sent 3 HTTP requests and 1287 bytes; received 19595741 bytes in response)
abort: HTTP request error (incomplete response)
(this may be an intermittent network failure; if the error persists,
consider contacting the network or server operator)

The mercurial client that comes with git bash (2.40.1)
is 5.0.2.

Might anyone have any idea how to fix this issue?

Thanks

Ed
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: About permission evaluation for repository group owner.

[Mads Kiilerich]

On 09/05/2023 16:04, toras wrote:
> I propose 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/dee1b60bad29621882eb769eb5bc8707647ccf1d 
.


As far as I have tried, I believe this change fixes the new owner to 
operate correctly. (Both from the web and from the API.)



Thanks for verifying.


> I propose 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/bf7369172810fb1a9452af767a2168edba3dc2f3


I believe that this change is also necessary to properly remove 
permissions from the previous owner.



Ok, then let's take this to the stable branch too.


> Do you see other problems related to these changes? Any other places 
where the code makes incorrect assumptions on repo groups

> and owner / permissions?

Related to the second issue, there seems to be a problem that "the 
owner (non-super user) of a group cannot set permissions for 
himself/herself".
In the permission settings screen, the owner cannot set the following 
write permissions for himself/herself.
Any attempt to do so fails with the message 'Cannot revoke permission 
for yourself as admin'.
I think this is part of the behavior that remains from when we were 
handling explicitly granting administrative privileges to groups.


However, some groups can be modified, and there may be conditions 
under which the above failure occurs.

This may be the case for groups created by ordinary users themselves.



Right - nice catch. I don't think there are any valid use cases for this 
code now. And there is also similar code in the web templates.


Please consider 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/ab8e9f05241a .



> (For some reason, repo group creation is more constrained in than 
repo creation... but that's yet another story.)

...
Sometimes I wonder why, because I want to create a group with the 
following structure, but cannot do so with only write permission.


personals <- Create by admin.
  + userA_group   <- Create by userA.
  + userB_group   <- Create by userB.



Yeah, if I remember correctly, it shows up in several places that repo 
group creation is considered more restricted than repos. For example, if 
I remember correctly, there is no way to allow ordinary users to create 
top level repo groups.


There could perhaps be some philosophical idea that deep nesting is bad, 
and that only admins should be allowed to add more complexity.


Or perhaps it is just that repo groups were added as a half-baked 
afterthought.


It could perhaps be changed, but that would be a different discussion, 
and not suitable for the stable branch.



/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: About permission evaluation for repository group owner.

[Mads Kiilerich]

On 07/05/2023 17:37, toras wrote:
Commit abc29122c7f2 has been addressed to allow repository group owner 
changes.

I think the owner change itself is working.
However, for non-admin users, the permission evaluation in the 
repository group seems to be incorrect.


For example, if you try to create a repository in that repository 
group as a changed owner user, you will get the error 'no permission 
to create repo in '.
After a little research, it seemed to me that the 
repository_group_permissions() in auth.py, which is used beyond the 
HasRepoGroupPermissionLevel() call, needs to be evaluated for being 
the owner of the repository group.

Could you please confirm this?



I think you are right.

Before, group ownership couldn't be used for anything, and everybody had 
to use explicit permissions on the group instead. On group creation, the 
owner is thus given explicit admin permissions unless it is a global 
admin. (That create problems if global admin permissions is removed from 
the user, and the user thus loses permissions for the groups they own.)


Recently, we made group ownership more manageable, but that also exposes 
that we have to make the implementation more complete. The new owner has 
to be given admin permissions somehow.


I think repository_group_permissions has to give admin permissions for 
the group owner, similar to how repository_permissions gives admin 
permissions to the repo owner. That shouldn't make the computation more 
complex or expensive, so that should be fine.


I propose 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/dee1b60bad29621882eb769eb5bc8707647ccf1d 
.


Also, we should stop giving explicit admin permissions on group 
creation. But that is a change with no immediate benefit, so that should 
probably happen on the stable branch.


I propose 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/bf7369172810fb1a9452af767a2168edba3dc2f3


Please, can you test these changes and verify they solve the problem for 
you?


Do you see other problems related to these changes? Any other places 
where the code makes incorrect assumptions on repo groups and owner / 
permissions?



Additionally, I have a question regarding the permission evaluation 
for repository groups, separate from the issue mentioned above.
Currently, regular users cannot create repositories within a 
repository group unless they have administrative privileges for the 
group.

I feel that requiring administrative privileges is a bit excessive.
What are your thoughts on this matter?



If you edit a repository group, the permissions tab will describe it as 
"Write" as "(Add repos)". Admin access should not be necessary. Please 
verify that you really see the behaviour you describe.


(For some reason, repo group creation is more constrained in than repo 
creation... but that's yet another story.)



/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] hg: add settings to make repositories non-publishing or use changeset evolution

[Mads Kiilerich]

On 20/04/2023 14:55, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1681995256 -7200
#  Thu Apr 20 14:54:16 2023 +0200
# Node ID 56aad162afcff89ec7e0b8cc7a60fff2bbde6ec1
# Parent  294f5a6814edc267c87a77dad5131c08a50fae1c
# EXP-Topic non_publishing_changeset_evolution
hg: add settings to make repositories non-publishing or use changeset evolution



Thanks. I'm absolutely positive to the publishing part of this change.

I have more concerns about the evolution part.

For that reason I would like to have it split up so they can land 
separately. While the code changes are very similar, they touch features 
that have very different maturity.



Setting repositories non-publishing and enabling changeset evolution makes it
possible to refine changesets collaboratively. Doing both at the same time will
therefore probably be the most common use case.

There are some users that push to-be-reviewed changesets to Kallithea
repositories, but don’t need or want obsolescence information to be shared.
Those users can now enable use of phases by setting the repository
non-publishing if they didn’t already do that by modifying the repository’s
hgrc.



I don't know about obsolescence information, but I think it is relevant 
to point out that this part of these changes is about controlling 
(disabling) phases.publishing , and that phases can be used as 
safetyguard independent of any use of evolve and obsolescence. Since 
repositories (and thus Kallithea) defaults to publishing and puts on 
safetyguards that can get in the way, it is nice to be able to disable 
it in the UI (and not just in the repo .hg/hgrc).




Having a publishing repository with changeset evolution can also be useful. For
example, there could be a project that has the committed changesets in a
publishing repository and uses email-based patch contributions. When a
maintainer commits a patch, possibly after some modifications, Mercurial has
the possibility to automatically add and share obsolescence information stating
that the committed changeset is a successor of the contributor’s original local
changeset that the patch was created from.



Regarding the evolution part of the change: I can see it (currently) is 
all about setting [experimental] evolution=all . That seems to be an 
undocumented Mercurial (or evolve?) flag, so it is really not clear what 
it does - both in general and for Kallithea operations in Web Ui and 
repo hosting. The relation to the unofficial evolve extension is not 
clear to me. The option is in the experimental namespace, so it doesn't 
seem like a thing to expose in the UI in a product that aim to be 
stable. It will presumably only be supported in some hg versions. It 
would be more relevant to support it in Kallithea if it was a proper 
Mercurial feature.


Some of these concerns can perhaps be addressed in the Kallithea UI or 
in commit message or code comments, but some of it would perhaps be 
better solved in upstream code and documentation that this change could 
reference.




Previously, there were no settings in the web interface that modify
per-repository Mercurial configs and no per-repository settings that apply to
hg repositories but not git repositories. Therefore, when adding these
settings, there was no pattern to follow and a lot of different design and
implementation choices could have been taken.



It is true that Kallithea so far doesn't have UI for per repo settings. 
It has always been possible to do this per repo by putting configuration 
inside the server side repos .hg/hgrc , or globally with manual entries 
in the Ui table.


One concern was how/where to store the info, in a way that scaled to all 
the things users possibly could want to set for each repo. Also, 
managing settings on each repo individually could be annoying. There 
could be very relevant use cases for settings on groups that were 
inherited to all repos inside the group.


Boolean fields in the Repository db records is probably a fine pragmatic 
solution.


BUT it will need a db migration step, per docs/dev/dbmigrations.rst . 
The new default repo settings also has to be considered - I'm not sure 
they would need migration steps too. (And this need for a manual db step 
do what we can't sneak this change in on the stable branch - it will 
have to go to the default branch.)




diff --git a/kallithea/controllers/api/api.py b/kallithea/controllers/api/api.py
--- a/kallithea/controllers/api/api.py
+++ b/kallithea/controllers/api/api.py
@@ -928,6 +928,8 @@
  landing_rev='rev:tip',
  enable_statistics=None,
  enable_downloads=None,
+publishing=None,
+enable_changeset_evolution=None,



This should probably be documented in api.rst .

Here I also notice a design decision: variables (and database fields 
etc) are named 'publishing', which seems a bit more generic than the 
current semantics of meaning exactly the 

Re: [PATCH 1 of 2 stable v3] tests: set Git author and committer name and email settings explicitly

[Mads Kiilerich]

On 20/04/2023 01:30, Manuel Jacob wrote:

On 20/04/2023 01.07, Mads Kiilerich wrote:

Thanks - this looks much more approachable.

But while processing v2 and our discussion and trying to give good 
input to what I would like to see in v3, I figured it perhaps would 
be better with a simpler solution.


What do you think about doing it as in 
https://kallithea-scm.org/repos/kallithea-incoming/changelog/61393fa29552/?size=2 
? (Possibly incorporating some of the new summary in v3?)


Implementation-wise, either would be fine for me, as long as the 
solution avoids system-dependent behavior (in that aspect, both 
implementations are equivalently good).


My changeset description is a bit more extensive. I’ll let you decide 
if merging the two patches is worth the effort in case you decide for 
your implementation.



Thanks for the collaboration. I pushed a mix of the two approaches.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH 1 of 2 stable v3] tests: set Git author and committer name and email settings explicitly

[Mads Kiilerich]

Thanks - this looks much more approachable.

But while processing v2 and our discussion and trying to give good input 
to what I would like to see in v3, I figured it perhaps would be better 
with a simpler solution.


What do you think about doing it as in 
https://kallithea-scm.org/repos/kallithea-incoming/changelog/61393fa29552/?size=2 
? (Possibly incorporating some of the new summary in v3?)


/Mads


On 20/04/2023 00:39, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1680121377 -7200
#  Wed Mar 29 22:22:57 2023 +0200
# Branch stable
# Node ID 846ca7f28bd40e07c76ed259ce96a31a85d0c4ea
# Parent  0a9ddb8cd8c117671ecaf2b4126c3eef09e80ce8
# EXP-Topic tests-git
tests: set Git author and committer name and email settings explicitly

Git tries to find out name and email in this order:

1. The author can be set e.g. via the `--author` option of `git commit`.
2. If set, the environment variables GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL,
GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL are taken.
3. If set, various config options are considered.
4. Unless disabled by the user.useconfigonly config, the names and emails are
infered from various system sources such as various fields from /etc/passwd,
/etc/mailname and the environment variable EMAIL.

The author was previously passed via (1), but that doesn’t work for the
committer.

We don’t modify Git’s configuration files, so the result of (3) depends on the
system the tests run on, which should be avoided. A follow-up patch will be
sent for instructing Git to not read the system Git configuration files.

(4) is also system-dependent. On my system, (4) was disabled in the Git
configuration. If I enabled it, Git tried to infer the committer name from a
field from /etc/passwd that is empty for my user I ran the tests on, which Git
didn’t like. The previous code passed the environment variable EMAIL, which,
according to a comment, is only required on some systems, but it’s unclear why.

By passing the names and emails via (2), we can set the author and committer
name and email uniformly and prevent Git from using the system-dependent ways
(3) and (4). The environment variables were introduced in 2005, so there should
be no backwards compatibility problems.

diff --git a/kallithea/tests/other/test_vcs_operations.py 
b/kallithea/tests/other/test_vcs_operations.py
--- a/kallithea/tests/other/test_vcs_operations.py
+++ b/kallithea/tests/other/test_vcs_operations.py
@@ -167,6 +167,27 @@
  return tempfile.mkdtemp(dir=base.TESTS_TMP_PATH, prefix=prefix, 
suffix=suffix)
  
  
+def _commit(vcs, dest_dir, message, *extra_args):

+email = 'm...@example.com'
+if os.name == 'nt':
+name = 'User'
+else:
+name = 'User ǝɯɐᴎ'
+
+return Command(dest_dir).execute(
+vcs,
+'commit',
+'-m',
+'"%s"' % message,
+*extra_args,
+HGUSER='%s <%s>' % (name, email),
+GIT_AUTHOR_NAME=name,
+GIT_AUTHOR_EMAIL=email,
+GIT_COMMITTER_NAME=name,
+GIT_COMMITTER_EMAIL=email,
+)
+
+
  def _add_files(vcs, dest_dir, files_no=3):
  """
  Generate some files, add it to dest_dir repo and push back
@@ -179,24 +200,10 @@
  open(os.path.join(dest_dir, added_file), 'a').close()
  Command(dest_dir).execute(vcs, 'add', added_file)
  
-email = 'm...@example.com'

-if os.name == 'nt':
-author_str = 'User <%s>' % email
-else:
-author_str = 'User ǝɯɐᴎ <%s>' % email
  for i in range(files_no):
  cmd = """echo "added_line%s" >> %s""" % (i, added_file)
  Command(dest_dir).execute(cmd)
-if vcs == 'hg':
-cmd = """hg commit -m "committed new %s" -u "%s" "%s" """ % (
-i, author_str, added_file
-)
-elif vcs == 'git':
-cmd = """git commit -m "committed new %s" --author "%s" "%s" """ % 
(
-i, author_str, added_file
-)
-# git commit needs EMAIL on some machines
-Command(dest_dir).execute(cmd, EMAIL=email)
+_commit(vcs, dest_dir, "committed new %s" % i, added_file)
  
  def _add_files_and_push(webserver, vt, dest_dir, clone_url, ignoreReturnCode=False, files_no=3):

  _add_files(vt.repo_type, dest_dir, files_no=files_no)
@@ -618,7 +625,7 @@
  # add submodule
  stdout, stderr = Command(base.TESTS_TMP_PATH).execute('git clone', 
fork_url, dest_dir)
  stdout, stderr = Command(dest_dir).execute('git submodule add', 
clone_url, 'testsubmodule')
-stdout, stderr = Command(dest_dir).execute('git commit -am "added 
testsubmodule pointing to', clone_url, '"', EMAIL=base.TEST_USER_ADMIN_EMAIL)
+stdout, stderr = _commit('git', dest_dir, "added testsubmodule pointing to %s" % 
clone_url, "-a")
  stdout, stderr = Command(dest_dir).execute('git push', fork_url, 
'master')
  
  # check for testsubmodule link in files page

___

Re: [PATCH 2 of 2 stable v2] tests: prevent Git system and global configuration from loading

[Mads Kiilerich]

On 19/04/2023 17:27, Manuel Jacob wrote:

On 19/04/2023 17.21, Manuel Jacob wrote:

On Arch Linux:

% git --version
git version 2.40.0
% GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null 
EMAIL='foo@bar' git commit -m "committed new 0" --author "User ǝɯɐᴎ 
"

Committer identity unknown




On CentOS 7:

% git --version
git version 2.36.5
% GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null 
EMAIL='foo@bar' git commit -m "committed new 0" --author "User ǝɯɐᴎ 
"

Committer identity unknown



But with openSUSE Leap 15.4 with git 2.35.3, it works (uses $USER as 
the name).


Same on openSUSE Tumbleweed with git 2.40.0.




That's interesting obvervations - thanks. It seems like we are far from 
a full understanding the problem we are trying to fix. There seems to be 
a high risk that we make more assumptions that will be invalid on some 
systems.


I verified that it works as I expect on my Fedora 38 system, both with 
the system git 2.40.0 and with a git build from 
https://github.com/git/git/ source . I don't really have access to other 
systems.


It could be somewhat interesting if you could try to build git from 
source on your failing systems. It could give a hint if it is some kind 
of distro patching or build configuration that makes a difference. Or if 
it is controlled by some other unknown factor.


It could also be interesting to ask some git experts if they could 
explain why for example 2.40.0 on Arch and openSUSE respond differently 
- that is really a question of reliable scripting of git and has nothing 
to do with Kallithea.


The code in https://github.com/git/git/blob/main/ident.c has many small 
steps and it is not obvious what could make it behave differently.


/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH 2 of 2 stable v2] tests: prevent Git system and global configuration from loading

[Mads Kiilerich]

On 19/04/2023 15:14, Manuel Jacob wrote:

On 19/04/2023 14.27, Mads Kiilerich wrote:

On 18/04/2023 20:35, Manuel Jacob wrote:
...
I think this changeset should come first. As you say, it makes the 
test work the same way everywhere. user.useconfigonly is probably 
just one of many settings that could break it.



If it still doesn't work for you with this change (without the 
previous change), it can't be because your setup has 
user.useconfigonly set. There must be some other explanation.


If I have only this changeset without the parent, command `git commit 
-m "committed new 0" --author "User ǝɯɐᴎ " 
"97u1nbm0setup.py"` fails with:


Committer identity unknown

*** Please tell me who you are.
...


Ok, after experimenting and reading git man pages, the shortest and 
clearest description seems to be:


The git commit --author option can tell git to use another name as 
author when committing, but git still has to know the user to use as 
committer (to be shown with --format=full).


Without any git user configuration (for example because there is no 
~/.gitconfig or because GIT_CONFIG_GLOBAL=/dev/null or because 
user.useConfigOnly), commit will fail with some kind of error, even if 
--author is specified.


But if EMAIL is set (and without user.useConfigOnly), it will use that 
as username, apparently combined with the getpwnam gecos information.



The following thus seems to works for me - no matter what global/system 
git configuration I have:


GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null EMAIL='foo@bar' 
git commit -m "committed new 0" --author "User ǝɯɐᴎ "


Silently dropping the setting of EMAIL in the first changeset is thus a 
bit risky. Could the consequences of that perhaps have interfered with 
your testing?


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH 2 of 2 stable v2] tests: prevent Git system and global configuration from loading

[Mads Kiilerich]

On 18/04/2023 20:35, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1680139355 -7200
#  Thu Mar 30 03:22:35 2023 +0200
# Branch stable
# Node ID e5251abd0a3c677d7bb0828f3a744789bd6fe4cb
# Parent  30082bb9719eb00f3be0081b7221d7c3061d4345
# EXP-Topic tests-git
tests: prevent Git system and global configuration from loading

This reduces differences between different testing environments. Something
similar is already done for Mercurial (in the lines directly above this
change).



Yes, I agree that it is a problem/bug that the global configuration is 
used at all. Global configuration should be disabled for git, as we do 
for hg.



The parent changeset has originally been added to support user.useconfigonly.
With this changeset, the original motivation for it becomes obsolete. However,
it is still necessary to set the committer name via a environment variable, at
least on my machine.



I think this changeset should come first. As you say, it makes the test 
work the same way everywhere. user.useconfigonly is probably just one of 
many settings that could break it.



If it still doesn't work for you with this change (without the previous 
change), it can't be because your setup has user.useconfigonly set. 
There must be some other explanation.


(Note that the GIT_CONFIG_ environment variables were introduced two 
years ago, and Kallithea still supports the 10 year old Git 1.7.4 . We 
must thus make sure we don't rely too much on the new feature.)



diff --git a/kallithea/tests/other/test_vcs_operations.py 
b/kallithea/tests/other/test_vcs_operations.py
--- a/kallithea/tests/other/test_vcs_operations.py
+++ b/kallithea/tests/other/test_vcs_operations.py
@@ -150,6 +150,8 @@
  testenv['LANGUAGE'] = 'en_US:en'
  testenv['HGPLAIN'] = ''
  testenv['HGRCPATH'] = ''
+testenv['GIT_CONFIG_SYSTEM'] = ''
+testenv['GIT_CONFIG_GLOBAL'] = ''



As mentioned before: the git man page is very clear that the variables 
should be set to /dev/null to skip reading configuration files.


There is no indication in the git man page (or anywhere else I have 
found) that setting to /dev/null shouldn't work across all platforms. It 
seems like git in compat/mingw.c takes care of making it cross platform.


There is no mention of the semantics of setting it to an empty string. I 
do not like relying on undefined behaviour.


If you disagree and have found a good reason to use an empty string 
instead of /dev/null, then please be explicit in code or commit message.



/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Timeout with git clone

[Mads Kiilerich]

Hi

The code in this area has worked surprisingly well since Kallithea 
inherited it, even though it has popped up regularly needing tricky 
maintenance. I agree it would be nice to refactor / reimplement this 
area. It is just that nobody invested time or sponsorship in doing it. I 
guess it hasn't caused enough pain for anybody to justify it ;-)


Yes, I agree that it probably would be much better to go back to use 
dulwich both for protocol serving and for providing data for the web 
frontend, instead of forking out to git. Disclaimer: I don't know has 
fast dulwich is these days. It could perhaps also be relevant to 
research what other python git hosting solutions do.


If interested in contributing in this area, a first step could be to 
create a proof of concept of switching back to Dulwich and doing some 
benchmarks - both for local cloning with infinite network bandwidth 
(where I doubt dulwich can match pure git) and for more realistic remote 
internet bandwidth (where I guess it doesn't matter much).


But also note that subprocessio no longer only is used by pygrack. It is 
also used for run_git_command in 
kallithea/lib/vcs/backends/git/repository.py (introduced in 
1f4d4b8d72f5), mainly for cloning and listing changesets. A full 
solution would require somehow replacing run_git_command with dulwich. 
But that can be done one at a time.


/Mads


On 18/04/2023 16:55, Quentin Wenger wrote:

Digging a bit deeper:

- The changeset that you linked 
(https://kallithea-scm.org/repos/kallithea/changeset/034e4fe1ebb2#rhodecodelibsubprocessiopy_n127)
 actually shows that historically it went the other way round, that is at first dulwich's 
server was used but then considered "buggy", therefore the implementation was 
replaced by some custom code.

- That custom code looks like coming from 
https://github.com/dvdotsenko/git_http_backend.py. That repo hasn't been 
updated since 2012, neither do its forks show any sign of recent activity.

- In contrast, dulwich, while officially still in beta, is actively developed.

IMhO the proper move would be to go back to dulwich. Chances are that those 
buggy things have been fixed in the last ten years. And if they haven't, better 
report them upstream than reinvent the wheel. By the way, do we have any more 
precise idea of what was considered buggy at the time?

What do you think?



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Freshly cloned Kallithea repository is corrupt

[Mads Kiilerich]

On 17/04/2023 15:45, Manuel Jacob wrote:

On 17/04/2023 13.39, Mads Kiilerich wrote:

Hi

That seems to be a Mercurial 6.4 regression, introduced by 
87f0155d68aa .


Can you confirm, and will you follow up upstream?


I can do that, but I would be interested in how you found out.



At first I couldn't reproduce. Then I found a setup where I could 
reproduce. I figured it could be a hg regression, and bisected hg.



/Mads



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Freshly cloned Kallithea repository is corrupt

[Mads Kiilerich]

Hi

That seems to be a Mercurial 6.4 regression, introduced by 87f0155d68aa .

Can you confirm, and will you follow up upstream?

/Mads


On 17/04/2023 03:10, Manuel Jacob wrote:

When I cloned the Kallithea repository, it was corrupted.

% hg clone https://kallithea-scm.org/repos/kallithea /tmp/kallithea
applying clone bundle from 
https://kallithea-scm.org/bundles/kallithea-b103f41a4954e2814dc547671771bdeebc1f80be.bzip2-v1.hg

adding changesets
adding manifests
adding file changes


added 5838 changesets with 24921 changes to 6256 files


finished applying clone bundle
searching for changes
adding changesets
adding manifests


adding file changes
added 2930 changesets with 10026 changes to 634 files (+1 heads)
56305 new obsolescence markers
new changesets abc1ada59076:0a9ddb8cd8c1
5838 local changesets published
updating to branch default
567 files updated, 0 files merged, 0 files removed, 0 files unresolved
% cd /tmp/kallithea
% hg summ
parent: 8765:b2c3d4568a5f
 Merge stable
branch: default
abort: 00manifest@47652662ac23c386e873c20b84f6aace9cca8bb5: no node
% hg up default
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
% hg up stable
abort: 00manifest@47652662ac23c386e873c20b84f6aace9cca8bb5: no node

I tried with both hg 6.4 and 6.4.1 (from different machines).

Disabling clone bundles (`--config ui.clonebundles=false`) fixed the 
problem.

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Timeout with git clone

[Mads Kiilerich]

Hi

I haven't seen that problem and can't reproduce it.

The wait for 10 seconds in some pretty obscure code came from a comment 
in 
https://kallithea-scm.org/repos/kallithea/changeset/034e4fe1ebb2#rhodecodelibsubprocessiopy_n127 
before The Big Fork. The comment became reality in 
https://kallithea-scm.org/repos/kallithea/changeset/01aca0a4f876#kallithealibvcssubprocessiopy_n125 
when moving to Python 3. It doesn't seem to have caused problems so far, 
but I might have been too naive and trusted the comment too much.


Does it work better for you if changing it back:

 kr.wait(2)

-    if not kr.wait(10):

+    if len(t) > ccm + 3:

 raise IOError(

 "Timed out while waiting for input from subprocess.")



I don't see why that should be a good change, but perhaps it fixes your 
issue.


Please let me know if you think I should push 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/35e5c3dcec22 .


/Mads


On 15/04/2023 01:21, Quentin Wenger wrote:

Hi,

When cloning a medium-sized repo (not extremely large but with a couple heavy 
media files), I consistently get a timeout preventing the cloning from 
completing.

Client:

$ git clonehttps://user@domain/main_website
Cloning into 'main_website'...
Password for 'https://user@domain':
remote: Enumerating objects: 10798, done.
remote: Counting objects: 100% (10798/10798), done.
remote: Compressing objects: 100% (5199/5199), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

The error occurs during the "Receiving objects:" phase, around 60%.

Server log with DEBUG:

2023-04-14 19:05:26.748 INFO  [kallithea.controllers.base] pull action on git repo 
"main_website" by "user" from IP
2023-04-14 19:05:26.748 DEBUG [kallithea.config.middleware.pygrack] handling 
cmd ['git', 'upload-pack', '--stateless-rpc', 
'/home/domain/hosting_kallithea/repos/main_website']
Exception in thread Thread-6:
Traceback (most recent call last):
   File "/opt/alt/python310/lib64/python3.10/threading.py", line 1016, in 
_bootstrap_inner
 self.run()
   File 
"/home/domain/hosting_kallithea/source/kallithea/lib/vcs/subprocessio.py", line 
129, in run
 raise IOError(
OSError: Timed out while waiting for input from subprocess.
[UID:1552][1444643] Child process with pid: 1444662 was killed by signal: 15, 
core dumped: no

Cloning via git+ssh directly instead of the https protocol works fine.

Has this been experienced before?
Is this just a matter of using a longer timeout value on line 128 of 
kallithea/lib/vcs/subprocessio.py? How was the value 10 seconds chosen in the 
first place? What about making it configurable if it is arbitrary?

Thanks,
Quentin
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: English in Accept-Language

[Mads Kiilerich]

Hi

You are right. That regressed with TurboGears 2.4.3, kind of documented 
on https://github.com/TurboGears/tg2/releases .


As a workaround, does it work as expected for you if setting
i18n.native = en
?

I will consider pushing 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/405e1d45700799bf8a2167420ae80544318f1022 
to stable - does that also work for you and look right?



Note that:
We already ship a catalog for English - that worked as workaround for 
2.4.2 .


And forcing a language by setting
i18n.native = en
only works if you also have
i18n.enabled=false

/Mads



On 07/04/2023 00:36, Quentin Wenger wrote:

Hi,

I'm new to Kallithea. I freshly installed an instance and got an issue with 
languages.

My browser sends the following header:
Accept-Language: en-US,en;q=0.7,fr-CH;q=0.3

I expect Kallithea to be rendered in English, but instead I see French.

I found in the mail archive that the issue has been discussed before:
https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003061.html
https://github.com/TurboGears/tg2/pull/115

So, what's the status? Has it changed since 2019? Are there workarounds known?
I tried:
- to init and compile a catalog for English
- to uncomment the `i18n.lang = en` line in my .ini
...but it didn't seem to fix the issue.

Any help appreciated!

Thanks and kind regards,
Quentin
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Transferring changeset approval status to rebased successors

[Mads Kiilerich]

Hi

Some brief comments to the big questions:

c.cs_repo.statuses() is already used for finding status for changeset 
hashes in bulk. It will perhaps also be able to handle that you pass all 
ancestor hashes for all pending PR changes. But you will of course have 
to process the result and pick the most recent approval to be the one 
that applies.


If you don't want to compute that when rendering web pages, it can also 
be computed in a push hook. (You can probably ignore the possibility of 
obsoleted changesets changing review status. Only the latest changeset 
will get new reviews from the web UI, and that will overrule any old 
result anyway.)


I guess you ideally also should verify that the changeset didn't change 
significantly since the previous approval. Perhaps by looking at the 
textual diff (without line numbers) and see if it is the same.


This seems to only be about one reviewer on each changeset. Great if 
that works for you. Doing the same for PRs with multiple reviewers with 
independent review status will be more tricky.


/Mads


On 25/03/2023 22:10, Manuel Jacob wrote:

Hi,

In one project I’m working on, we do code review of single changesets 
in a feature branch (usually the changesets are quite small and on 
average more than 10 are submitted for review at the same time). We 
also use Mercurial’s changeset evolution quite heavily. Feature 
branches are rebased regularly and single changesets are amended 
between two reviews (causing the descendants of these changesets to be 
rebased by the evolve extension).


Currently, we track the review status of each of these changesets 
manually. After the branch is rebased, each of the rebased changesets 
is shown as unreviewed in Kallihea. It would be a significant 
improvement if Kallithea showed for each changeset whether an 
“unchanged” predecessor was already approved.


Thanks to the obsoleteness markers provided by Mercurial, this is easy 
to determine. The algorithm would walk through the predecessors if 
there is only one and only the parent changed in between them, until 
it hits a changeset whose status is not “unreviewed”.


One question is how to show this information to the user. What would 
work for me is to show "approved predecessor" in all places where 
"approved" can be shown. Instead of a green circle, it could show the 
outline of a green circle. (The same could be applied to “under 
review” and “not approved”).


Another question is when to run the logic. Running it each time the 
review status is shown somewhere would work good enough for us. 
Caching this is not easy. It would need to be invalidated each time a 
predecessor is added or its review status is changed. Recomputing it 
each time shouldn’t be a problem in practice because the obsoleteness 
markers are stored in-memory, the number of considered predecessors is 
limited (until the algorithm hits a “changed” or already reviewed 
predecessor) and in most places where the review status is shown, the 
changeset description is also shown, which has to be read from disk, 
so walking the predecessors should not contribute much to the total time.


What do you think?
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] api: include creation date in comments

[Mads Kiilerich]

Thanks - pushed to stable.

/Mads



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH 2 of 2] api: add possibility to optionally return comments from get_changeset()

[Mads Kiilerich]

Thanks - pushed to stable.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH 2 of 2] tests: prevent Git system and global configuration from loading

[Mads Kiilerich]

On 30/03/2023 03:37, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1680139355 -7200
#  Thu Mar 30 03:22:35 2023 +0200
# Node ID 4933600c49883dfc142f97a32dd08f9dd82b0d31
# Parent  2304f8da62b03f6cfd966b72eec70c38c4d52e37
# EXP-Topic tests-git
tests: prevent Git system and global configuration from loading

This reduces differences between different testing environments. Something 
similar is already done for Mercurial (in the lines directly above this change).

The parent changeset has originally been added to support user.useconfigonly. 
With this changeset, the original motivation for it becomes obsolete. However, 
it is still necessary to set the committer name via a environment variable, at 
least on my machine.

diff --git a/kallithea/tests/other/test_vcs_operations.py 
b/kallithea/tests/other/test_vcs_operations.py
--- a/kallithea/tests/other/test_vcs_operations.py
+++ b/kallithea/tests/other/test_vcs_operations.py
@@ -150,6 +150,8 @@
  testenv['LANGUAGE'] = 'en_US:en'
  testenv['HGPLAIN'] = ''
  testenv['HGRCPATH'] = ''
+testenv['GIT_CONFIG_SYSTEM'] = ''
+testenv['GIT_CONFIG_GLOBAL'] = ''



Looking at the git man page, I wonder why not using /dev/null?

(I don't like how Mercurial put different semantics to undefined and 
empty environment variables. That used to cause problems on some 
almost-extinct unices. But it seems to work...)



/Mads



  testenv.update(environ)
  p = Popen(command, shell=True, stdout=PIPE, stderr=PIPE, 
cwd=self.cwd, env=testenv)
  stdout, stderr = p.communicate()

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH 1 of 2] tests: set Git author and committer name and email settings explicitly

[Mads Kiilerich]

On 30/03/2023 03:37, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1680121377 -7200
#  Wed Mar 29 22:22:57 2023 +0200
# Node ID 2304f8da62b03f6cfd966b72eec70c38c4d52e37
# Parent  7037365a7bc351b81f05c790c6d3417d81d7bd5d
# EXP-Topic tests-git
tests: set Git author and committer name and email settings explicitly

Passing at least GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL as environment 
variables is necessary on my machine, which has the user.useconfigonly config 
set. The author could be passed via command-line options, but it seems best to 
pass everything uniformly.



(We usually wrap commit messages at less than 80 columns.)


I agree with the intent here.

But EMAIL=email is lost?

It is slightly annoying to have to introduce global variables. It would 
be slightly better to have a _commit_files() function that apply 
environment variables directly to execute() and could be invoked in both 
places. But that is not a deal breaker.


/Mads



diff --git a/kallithea/tests/other/test_vcs_operations.py 
b/kallithea/tests/other/test_vcs_operations.py
--- a/kallithea/tests/other/test_vcs_operations.py
+++ b/kallithea/tests/other/test_vcs_operations.py
@@ -167,6 +167,23 @@
  return tempfile.mkdtemp(dir=base.TESTS_TMP_PATH, prefix=prefix, 
suffix=suffix)
  
  
+email = 'm...@example.com'

+if os.name == 'nt':
+name = 'User'
+else:
+name = 'User ǝɯɐᴎ'
+
+HG_USER = '%s <%s>' % (name, email)
+# If the user.useconfigonly config is set, Git won't try to auto-detect the
+# name and email. For this case, we need to pass them as environment variables.
+GIT_NAME_AND_EMAIL_ENV_VARS = dict(
+GIT_AUTHOR_NAME=name,
+GIT_AUTHOR_EMAIL=email,
+GIT_COMMITTER_NAME=name,
+GIT_COMMITTER_EMAIL=email,
+)
+
+
  def _add_files(vcs, dest_dir, files_no=3):
  """
  Generate some files, add it to dest_dir repo and push back
@@ -179,24 +196,18 @@
  open(os.path.join(dest_dir, added_file), 'a').close()
  Command(dest_dir).execute(vcs, 'add', added_file)
  
-email = 'm...@example.com'

-if os.name == 'nt':
-author_str = 'User <%s>' % email
-else:
-author_str = 'User ǝɯɐᴎ <%s>' % email
  for i in range(files_no):
  cmd = """echo "added_line%s" >> %s""" % (i, added_file)
  Command(dest_dir).execute(cmd)
  if vcs == 'hg':
  cmd = """hg commit -m "committed new %s" -u "%s" "%s" """ % (
-i, author_str, added_file
+i, HG_USER, added_file
  )
  elif vcs == 'git':
-cmd = """git commit -m "committed new %s" --author "%s" "%s" """ % 
(
-i, author_str, added_file
+cmd = """git commit -m "committed new %s" "%s" """ % (
+i, added_file
  )
-# git commit needs EMAIL on some machines
-Command(dest_dir).execute(cmd, EMAIL=email)
+Command(dest_dir).execute(cmd, **GIT_NAME_AND_EMAIL_ENV_VARS)
  
  def _add_files_and_push(webserver, vt, dest_dir, clone_url, ignoreReturnCode=False, files_no=3):

  _add_files(vt.repo_type, dest_dir, files_no=files_no)
@@ -618,7 +629,7 @@
  # add submodule
  stdout, stderr = Command(base.TESTS_TMP_PATH).execute('git clone', 
fork_url, dest_dir)
  stdout, stderr = Command(dest_dir).execute('git submodule add', 
clone_url, 'testsubmodule')
-stdout, stderr = Command(dest_dir).execute('git commit -am "added 
testsubmodule pointing to', clone_url, '"', EMAIL=base.TEST_USER_ADMIN_EMAIL)
+stdout, stderr = Command(dest_dir).execute('git commit -am "added 
testsubmodule pointing to', clone_url, '"', **GIT_NAME_AND_EMAIL_ENV_VARS)
  stdout, stderr = Command(dest_dir).execute('git push', fork_url, 
'master')
  
  # check for testsubmodule link in files page

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: username allowed characters

[Mads Kiilerich]

Hi

Well ... since you use @ in usernames, I think you *do* have an issue 
with the way Kallithea assumes emails and usernames are different 
"namespaces" (or rather "value spaces").


I'm reluctant to allow @ in usernames in general without making sure 
that it works correctly in all cases, without opening up for new problems.


As mentioned, one case that must be considered is when the username of 
one user is the email of another user. Some invariant would have to be 
enforced.


We need a convincing story of how get_by_username_or_email and all the 
invocations of it should handle emails as username. And there might be 
others.


It seems like in your setup, both kallithea/controllers/base.py and 
kallithea/lib/auth_modules/__init__.py will find the corresponding user 
object with get_by_username_or_email ... which will look up the user by 
matching on the email field. You say that "the mail address is still a 
different variable, but probably with the same content" ... but 
apparently it *must*  have the same content for your setup to work 
correctly. But then I don't understand what you say about the case where 
Email is different than the login.


Another potential problem that comes to mind is auto completion of 
usernames, for example in @mention. There might not be a problem there, 
but it would have to be verified.


So while it is great that it works for you with this minor change, I 
don't think it is a setup that can be recommended or that we would 
consider "supported". Fixing it properly would imply investing in bigger 
changes to the fragile and critical authentication system while making 
sure nothing breaks for other use cases.


For upstream Kallithea, I think it would be more feasible to expand ldap 
auth to allow login with email, but still use a "real" username without 
@ inside Kallithea ... assuming your ldap has another field that can be 
used for that purpose. But I don't know if that could work for you and 
with your IT department requirements.


/Mads


On 22/02/2023 10:34, s...@telenet.be wrote:

Hi Mads,

I don't have issues with emails and usernames in different namespaces. 
It is not my intention to use the "username" as email address, lets 
keep them indeed separate. It is just problematic that the username in 
my case also looks like an email address (with '@')


As it happens, our IT-dept has a hard request to use the email to 
login in systems, so I need to use the email as "Login Attribute" for 
LDAPS (the AD attribute is called userPrincipleName but I don't know 
if that is a standard), no problem here. The "Email Attribute" just 
gets the same data, so for Kallithea, the mail address is still a 
different variable, but probably with the same content.
To complicate things further, this 'userPrincipleName' can be an 
email-address on a different domain, than the standard 'Email 
Attribute' for a specific user, in this case the Email is different 
than the login.


The issue arises when I want to change a setting on a user (for 
example make someone admin), then the earlier 'valid' username is not 
valid anymore. Hence, the little hack, just allow a slightly more 
flexible rule to validate the username.


Cheers,

Stefaan




*Van: *"Mads Kiilerich" 
*Aan: *"vanheesbeke stefaan" , "kallithea-general" 


*Verzonden: *Dinsdag 21 februari 2023 15:25:10
*Onderwerp: *Re: username allowed characters

Hi

Thank you.

But ...

Kallithea generally tries to keep logins and emails unambiguous and in 
different namespaces. It is arguably a bug if Kallithea allows LDAP to 
use email addresses as usernames.


The assumption is probably not entirely enforced, but it and the 
consequences show up for example in get_by_username_or_email .


If allowing @ in usernames, it should perhaps be enforced that it only 
is allowed if it matches the email ... but that seems like a hack that 
would be hard to enforce and not really feasible.


But it is already generally possible to login with email instead of 
username. Perhaps that doesn't work with LDAP? Can you set attr_login 
to point at the email attribute? Or does that have other bad 
consequences? Something that could be fixed instead?


/Mads


On 21/02/2023 12:03, s...@telenet.be wrote:

Hello,

I'm using LDAP to authenticate users to our system. Out IT-dept
hammers on the fact that we need to use the email-address of users
to login.

The login works ok, but when I want to change the settings for a
certain user, it complains about a '@' in the user name.A simple
patch during docker build solved this issue.

Since email addresses are used regularly for logging in, maybe
this can be also in the next version of Kallithea.


Simply s

Re: username allowed characters

[Mads Kiilerich]

Hi

Thank you.

But ...

Kallithea generally tries to keep logins and emails unambiguous and in 
different namespaces. It is arguably a bug if Kallithea allows LDAP to 
use email addresses as usernames.


The assumption is probably not entirely enforced, but it and the 
consequences show up for example in get_by_username_or_email .


If allowing @ in usernames, it should perhaps be enforced that it only 
is allowed if it matches the email ... but that seems like a hack that 
would be hard to enforce and not really feasible.


But it is already generally possible to login with email instead of 
username. Perhaps that doesn't work with LDAP? Can you set attr_login to 
point at the email attribute? Or does that have other bad consequences? 
Something that could be fixed instead?


/Mads


On 21/02/2023 12:03, s...@telenet.be wrote:

Hello,

I'm using LDAP to authenticate users to our system. Out IT-dept 
hammers on the fact that we need to use the email-address of users to 
login.


The login works ok, but when I want to change the settings for a 
certain user, it complains about a '@' in the user name.A simple patch 
during docker build solved this issue.


Since email addresses are used regularly for logging in, maybe this 
can be also in the next version of Kallithea.



Simply said : just adding @ to the regexp for username does the job.

--- validators.py    2023-02-21 10:25:27.657212999 +
+++ validators_new.py    2023-02-21 10:26:40.560218089 +
@@ -92,7 +92,7 @@
 msg = self.message('username_exists', state, 
username=value)

 raise formencode.Invalid(msg, value, state)

-    if re.match(r'^[a-zA-Z0-9\_]{1}[a-zA-Z0-9\-\_\.]*$', 
value) is None:
+    if re.match(r'^[a-zA-Z0-9\_]{1}[a-zA-Z0-9\-\_\.@]*$', 
value) is None:

 msg = self.message('invalid_username', state)
 raise formencode.Invalid(msg, value, state)
 return _validator

Cheers,
Stefaan


BTW : this is the best package I encountered since bitbucket stopped 
with Hg.



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] pullrequests: introduce limit to stop displaying additional changes

[Mads Kiilerich]

Thank you. Pushed to stable with some minor tweaks and additions.

/Mads


On 18/01/2023 07:45, Mathias De Mare wrote:

# HG changeset patch
# User Mathias De Mare 
# Date 1672834977 -3600
#  Wed Jan 04 13:22:57 2023 +0100
# Branch stable
# Node ID ac278c9c011136b72de43d8dbd742f9cf3dbf020
# Parent  b7efb8fdc45fb95ce8ece04e1aed7d965c300bae
pullrequests: introduce limit to stop displaying additional changes

The previous pull request threw away some of the changesets
to keep the total amount more manageable, but this results
in an (for the user) unpredictable subset of changesets being shown.

To resolve this issue, we instead do not display any additional changes
if the amount of additional changes is larger than a user-defined limit.

This could be extended further with "too long to be shown - click here
to show", but that was quite a bit of additional work
and did not cover our use case of not allowing the display at all
in case of too many additional changes.

diff --git a/kallithea/controllers/pullrequests.py 
b/kallithea/controllers/pullrequests.py
--- a/kallithea/controllers/pullrequests.py
+++ b/kallithea/controllers/pullrequests.py
@@ -35,6 +35,8 @@ from tg import tmpl_context as c
  from tg.i18n import ugettext as _
  from webob.exc import HTTPBadRequest, HTTPForbidden, HTTPFound, HTTPNotFound
  
+import kallithea

+
  import kallithea.lib.helpers as h
  from kallithea.controllers import base
  from kallithea.controllers.changeset import create_cs_pr_comment, 
delete_cs_pr_comment
@@ -494,6 +496,8 @@ class PullrequestsController(base.BaseRe
  except IndexError: # probably because c.cs_ranges is empty, probably 
because revisions are missing
  pass
  
+rev_limit = safe_int(kallithea.CONFIG.get('next_iteration_rev_limit'), 0)

+
  avail_revs = set()
  avail_show = []
  c.cs_branch_name = c.cs_ref_name
@@ -563,9 +567,16 @@ class PullrequestsController(base.BaseRe
  except ChangesetDoesNotExistError:
  c.update_msg = _('Error: some changesets not found when 
displaying pull request from %s.') % c.cs_rev
  
-c.avail_revs = avail_revs

-c.avail_cs = [org_scm_instance.get_changeset(r) for r in avail_show]
-c.avail_jsdata = graph_data(org_scm_instance, avail_show)
+if rev_limit and len(avail_revs) > rev_limit:
+c.update_msg = _('Additional changesets (%d) are not shown because 
they exceed the limit (%d).') % (len(avail_revs), rev_limit)
+c.avail_revs = []
+c.avail_cs = []
+c.avail_jsdata = None
+else:
+c.rev_limit_reached = False
+c.avail_revs = avail_revs
+c.avail_cs = [org_scm_instance.get_changeset(r) for r in 
avail_show]
+c.avail_jsdata = graph_data(org_scm_instance, avail_show)
  
  raw_ids = [x.raw_id for x in c.cs_ranges]

  c.cs_comments = c.cs_repo.get_comments(raw_ids)

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Patch for repo groups API documentation

[Mads Kiilerich]

(Please always send to the public mailing list - not just private.)

Thanks. I applied the patch to the stable branch, with some trivial 
changes for consistency.


(Note that get_repo_group already had a docstring in api.py that is a 
bit more verbose and possibly better. I'm keeping an eye on making these 
converge and making it clear which one is better.)


/Mads


On 27/12/2022 19:38, Louis Bertrand wrote:

Hello Mads,
I'm catching up on a promised update to the API docs for repo groups. Please 
see attached patch for docs\api\api.rst
Hope this helps
  --Louis

--
Louis Bertrand, P.Eng.
Professor, Faculty of Science, Engineering and Information Technology
Durham College

-Original Message-----
From: Mads Kiilerich 
Sent: October 29, 2022 12:08 PM
To: Louis Bertrand 
Subject: Re: Patch for repo groups API documentation

[EXTERNAL EMAIL - USE CAUTION]

(Private reply)

Thanks. But the patch seems to be for your changes combined with a merge. That 
is hard to apply here.

Can you push the changes instead, using something like hg push -f -r 
8b9e6a60ece02ca505d32e48f5fb19829ffc5490
https://kiile...@kallithea-scm.org/repos/kallithea-incoming
<>

That should hopefully make it possible for me to untangle and extract your 
changes cleanly.

(But you are also welcome to clean it up and send in other ways if you
prefer.)

/Mads


On 29/10/2022 15:42, Louis Bertrand wrote:

Hi,
I started documenting the repository group API. Here's a patch, but it might be 
messy because of conflicts due to whitespace adjustments while I was working on 
it.
Let me know if it's a problem and I'll base the changeset on a more recent 
version.
All the best
   --Louis

--
Louis Bertrand, P.Eng.
Professor, Faculty of Science, Engineering and Information Technology
Durham College




This message is intended only for the named recipients. This message may 
contain information that is confidential or exempt from disclosure under 
applicable law. Any dissemination or copying of this message by anyone other 
than a named recipient is strictly prohibited. If you are not a named recipient 
or an employee or agent responsible for delivering this message to a named 
recipient, please notify us immediately, and permanently destroy this message 
and any copies you may have. Warning: Email may not be secure unless properly 
encrypted.

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general




___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: About API code comments and some behavior.

[Mads Kiilerich]

On 18/12/2022 10:48, toras wrote:

>> Attachment: fix-update_repo_group-2.patch
>
> Lots of things happening in that patch. Can you explain in more 
details what is changed and why ... and perhaps do it in several
> simpler changesets? 

...

The second is to modify the log output.
Sorry for the confusion, the change to the f string was just for my 
own viewing pleasure, not related to the issue.



Ok. But I will just start out by making this piece of code use lazy %s 
expansion for logging like we do in all other places.



The entities enumerated by recursive_groups_and_repos() to update the 
full path of the child elements are processed with log output, but the 
starting repository group has already had its entities updated before 
the loop.
So, for example, updating the name from 'GroupA' to 'GroupB' will 
result in the following log output

(Only the origin entity. Child elements are correct logs.)

  [kallithea.model.repo_group] Fixing group GroupB to new name GroupB

The following logs should be correct.

  [kallithea.model.repo_group] Fixing group GroupA to new name GroupB

For the purpose of logging the name before the change, the patch 
outputs the log before updating the entity and skips the originating 
entity in a loop process.



Right - thanks for clarifying. I applied fix doing just that, but also 
made sure we don't do excessive logging when parameters specified 
without changing.



Thanks for the contribution.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Kallithea Setup Docs

[Mads Kiilerich]

Hi David

To get more precise communication and avoid further confusion, please 
edit and repost the message after clarifying:


* when saying "uwsgi", make it clear if referring to the uWSGI software 
in general, or the uwsgi protocol


* only say "uwsgi server" when referring to the uWSGI software in 
general, not necessarily serving the uwsgi protocol


* say something like "uwsgi protocol server" if the point is the 
protocol and not the software


* (the current Kallithea uWSGI "support" is best described as "uwsgi 
http server")


/Mads



On 12/12/2022 19:00, David Griffin wrote:

Hi Mads,

I think there may be some misconceptions about uwsgi. uwsgi appears to 
be designed as something more akin to a runtime for uwsgi 
applications, which then interacts with a compatible webserver. If you 
were hosting multiple uwsgi applications, then it would normally be 
preferable to have each one be hosted on its own uwsgi server. The 
webserver then redirects to each of the uwsgi application servers as 
required. Therefore I don't agree that Kallithea could not supply a 
useful uwsgi server configuration file, because the configuration file 
only has to describe how to run Kallithea. If someone wants to run 
multiple uwsgi applications, they should most likely be running 
multiple uwsgi servers (even if those servers are just running on 
different sockets of the same machine) - if they don't, they're giving 
up a lot uwsgi's scalable design choices. Similarly, given uwsgi's 
nature as something like a runtime, I'd argue that running a uwsgi 
server is quite a lot simpler than some of the instructions you have 
on your setup page because Kallithea can offload all the interfacing 
to the actual webserver to uwsgi, regardless of exactly what that 
server is.


Therefore, taking into account what you've said, as well as my own 
research into the topic, I think my specific proposed change would be 
to change the "preferred" method to run Kallithea behind another 
webserver to be via uwsgi. This has a bunch of positives for 
maintaining Kallithea:


1) It follows best practice for deploying uwsgi apps. Your docs have 
an example of running behind nginx with http forwarding, which is not 
an ideal way of running a uwsgi app.
2) It offloads the integration with web servers to the uwsgi project, 
meaning that if something changes upstream, Kallithea doesn't need to 
update its instructions / way of doing thigns. For example, for 
Apache, mod_uwsgi has fallen out of favour and mod_proxy_uwsgi seems 
to be preferred, or at least according to the uwsgi docs. (Note: this 
also means that Apache no longer has special instructions for running 
uwsgi applications)
3) Similar to the previous point, this would expand support to other 
web servers without needing any extra effort in Kallithea.
4) Also similarly to 3, this would simplify the documentation - 
Kallithea would only need to document setting up the uwsgi app, and 
then point users to the uwsgi docs for integrating the uwsgi app with 
their preferred webserver. This would substantially reduce the size of 
the setup instructions, being able to remove all sections on specific 
servers (i.e. Apache, nginx), and thus reduce the maintenance burden.
5) This would address the potential confusion between uwsgi as an HTTP 
server and uwsgi as a uwsgi server by adding a simple note to the HTTP 
instructions that if the user wants to run behind an HTTP server, they 
should follow the uwsgi server instructions instead.
6) Potential to remove untested/unused templates from the codebase, as 
there would be a preferred method to replace them.


Doing this would require mostly changes to the documentation, I think. 
The only potential change to the code might be the addition of a uwsgi 
server setup template for config-create, which might require a little 
bit of work, as well as the potential removal of any untested / 
unnecessary templates. If this (or some variant after feedback) seems 
like a good idea, I'd be happy to spend some time on it.


One aside: manage-script-name seemed to be necessary in my setup. Some 
of the environ variables that Kallithea depends on (If memory serves, 
PATH_INFO) were not being set at all, which obviously broke things. 
However, while setting manage-script-name fixed the issue, I'm not 
entirely sure if the issue was caused by lighttpd not following the 
uwsgi spec correctly - this is something I should perhaps test when I 
can. As far as uwsgi-socket goes, it seems to be just a synonym for 
socket.


All the best,
- David

On Mon, 12 Dec 2022 at 15:12, Mads Kiilerich  wrote:

Hi David

The Kallithea docs aim to help getting a very basic setup with the
essentials. Something that perhaps can be used directly, but mainly
serve as a starting point for further setup which is outside the
scope
of Kallithea. It is import

Re: About API code comments and some behavior.

[Mads Kiilerich]

On 30/10/2022 13:00, toras wrote:

> Yes. Can you confirm this will solve the code problems you reported:
> 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/088551a24485247af328f0b8e395fdbbcd62a700 
?


1. Modify create_repo()
  I don't think this change is correct.
  I overwrote api.py and called create_repo, but 
enable_downloads/enable_statistics seems to have no effect.


  I think the call flow during API execution is as follows.

  api.py : ApiController.create_repo()
    repo.py : RepoModel.create()
  repo.py : (@celerylib.task) create_repo()

  I think that the third function does not refer to form_data, but 
refers to defs to obtain the default value and use it.

https://kallithea-scm.org/repos/kallithea/files/2dd317e9cc4b5ff94c1ddae7ee772d20b61259a9/kallithea/model/repo.py#L717
  I think this is probably implemented in conjunction with the web 
front end.


  I see two options for changes to the create_repo API.
  I think this requires a policy decision.

  Scenario 1:
    The API will no longer accept that parameter.
    Attachment: 1-scenario1.patch

  Scenario 2:
    Support parameter specification in RepoModel.
    Attachment: 1-scenario2.patch



Right. Thanks. I will go with scenario2 ... but take it further and make 
repo_enable_downloads/repo_enable_statistics mandatory in the internal 
API and expose them in the UI form as well. As a part of that, I will 
also add better test coverage and clean up the UI forms.




2. Modify update_repo_group()
  I think there are two issues with parent handling.

  The first is.
  Other api parameters can basically target entities by name.
  It seems to me that as the parent_group_id parameter to 
RepoGroupModel.update(), the API parameter should be passed with name 
resolution instead of passing it as is.

    Attachment: 2-parent_id.patch



Agreed. I will do that ... but with slight changes.



  The second is still unresolved and not yet known in detail.
  The problem is that changing the parent group via the API will leave 
the database and file system in an inconsistent state.
  At that time, new_path does not seem to indicate the correct path in 
RepoGroupModel.update(). So the folder is not moved.
  It seems correct when run from the web front end. I still don't know 
what the difference is.


...

I have found the cause of the problem with the update_repo_group API 
behavior that I wrote about in my previous email.
The implementation of RepoGroupModel.update() does not seem to work 
correctly unless it includes a valid group_name in args, even if the 
name is not changed.
If no new name is specified for the API parameter, it may be necessary 
to pass the original name.
I have created a patch to api.py based on 088551a24485 from 
kallithea-imcoming.

    Attachment: fix-update_repo_group-1.patch


Agreed, but I think the problem is in RepoGroupModel.update , where it 
failed to update group_name to the new full path path after moving *if* 
group_name wasn't specified. I will fix that instead.



Also, I thought it was not good to have inconsistent results depending 
on the call parameters, so I refactored the implementation of 
RepoGroupModel.update().
It also includes changes that correct incorrect logging output. 
However, this is not required for API operation.

    Attachment: fix-update_repo_group-2.patch


Lots of things happening in that patch. Can you explain in more details 
what is changed and why ... and perhaps do it in several simpler changesets?


For example:

What inconsistent results did you see?

What logging was incorrect? (But to avoid pointless formatting of log 
strings that will be ignored because of the logging level, we generally 
prefer to avoid using log.debug('%s' % x) but prefer log.debug('%s', x) .)


Also note that repo_group.parent_group is a convenience thing in the 
Object Relationship Mapper to automatically fetch whatever 
repo_group.parent_group_id refers to. When one of them are changed, the 
other one should change too.




Supplementation.
It may not be necessary, but I will publish the docker files and 
scripts that I used in the confirmation work.

https://github.com/toras9000/kallithea_api_test



Thank you. There are so many ways to do Docker. Great that your setup 
can cover this and provide additional testing compared to what is in the 
upstream repo.



I took a bit of a chance and pushed the changes to the stable branch. I 
hope you can confirm it works for you now.


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] kallithea pullrequests: allow limiting the number of additional changes shown

[Mads Kiilerich]

Thank you.

Is the overwhelming on the server side or in the browser?

I don't like showing incorrect information. Lists must be reliable. It 
must be made quite clear when lists have been truncated.


It would conceptually also be nice to be able to opt out of the truncation.

But that raise the questions: Why truncate from one end and not the 
other? Could it be possible to give preference to truncate merged 
changesets before truncating descendants? Doing further computation on a 
truncated list certainly feels wrong. And why bother trying to show any 
list at all if it only shows a random subset and thus can't be used for 
much?


Would it be a problem for your usecase to compute the full revision 
list, check the amount of revisions and 
request.GET.get('show_additional_changes'), and conditionally show "too 
long to be shown - click here to show" instead? (More or less like how 
fulldiff is handled.)


Also: To minimize dependencies on tg, we prefer to consistently use 
kallithea.CONFIG instead of tg.config (see kallithea/config/app_cfg.py ).


/Mads


On 05/12/2022 17:18, Mathias De Mare wrote:

# HG changeset patch
# User Mathias De Mare 
# Date 1669726298 -3600
#  Tue Nov 29 13:51:38 2022 +0100
# Node ID 2a5de196e40454dc9437e31e2224c183779a61d9
# Parent  7037365a7bc351b81f05c790c6d3417d81d7bd5d
kallithea pullrequests: allow limiting the number of additional changes shown

We've noticed some scalability issues when many descendants exist
for the changesets in a pull request.
If we limit the number of proposed changesets to add to the review,
we no longer overwhelm Kallithea in this case.

(This occurred because we were merging a lot of heads in the repository.)

diff --git a/kallithea/controllers/pullrequests.py 
b/kallithea/controllers/pullrequests.py
--- a/kallithea/controllers/pullrequests.py
+++ b/kallithea/controllers/pullrequests.py
@@ -30,6 +30,7 @@ import traceback
  
  import formencode

  import mercurial.unionrepo
+import tg
  from tg import request
  from tg import tmpl_context as c
  from tg.i18n import ugettext as _
@@ -68,6 +69,14 @@ def _get_reviewer(user_id):
  
  return user
  
+def _filter_additional_changes_revs(revs):

+"""Allow hooking in to filter out some of the additional changes."""
+additional_changes_rev_limit = 
tg.config.get('additional_changes_rev_limit')
+if additional_changes_rev_limit and 
additional_changes_rev_limit.isnumeric():
+new_revs = list(revs)
+return new_revs[-int(additional_changes_rev_limit):]
+else:
+return revs
  
  class PullrequestsController(base.BaseRepoController):
  
@@ -523,6 +532,7 @@ class PullrequestsController(base.BaseRe

  else: # look for descendants of PR head on source branch in 
org repo
  avail_revs = org_scm_instance._repo.revs('%s:: & 
branch(%s)',
   revs[0], 
c.cs_branch_name)
+avail_revs = _filter_additional_changes_revs(avail_revs)
  if len(avail_revs) > 1: # more than just revs[0]
  # also show changesets that not are descendants but 
would be merged in
  targethead = 
other_scm_instance.get_changeset(c.a_branch_name).raw_id
@@ -537,6 +547,7 @@ class PullrequestsController(base.BaseRe
  hgrepo = org_scm_instance._repo
  show = set(hgrepo.revs('::%ld & !::parents(%s) & 
!::%s',
 avail_revs, revs[0], 
targethead))
+show = _filter_additional_changes_revs(show)
  if show:
  c.update_msg = _('The following additional 
changes are available on %s:') % c.cs_branch_name
  else:
diff --git a/kallithea/tests/functional/test_pullrequests.py 
b/kallithea/tests/functional/test_pullrequests.py
--- a/kallithea/tests/functional/test_pullrequests.py
+++ b/kallithea/tests/functional/test_pullrequests.py
@@ -1,7 +1,9 @@
  import re
  
+import mock

  import pytest
  
+import kallithea.controllers

  from kallithea.controllers.pullrequests import PullrequestsController
  from kallithea.model import db, meta
  from kallithea.tests import base
@@ -390,3 +392,18 @@ class TestPullrequestsGetRepoRefs(base.T
  content='line1\n', message='commit1', vcs_type='hg',
  parent=None, newfile=True)
  # TODO
+
+class TestPullrequests(base.TestController):
+
+def test_filter_additional_changes_no_rev_limit(self):
+config_mock = {
+}
+with mock.patch('tg.config', config_mock):
+assert len(kallithea.controllers.pullrequests._filter_additional_changes_revs(["1", 
"2", "3", "4"])) == 4
+
+def test_filter_additional_changes_rev_limit(self):
+config_mock = {
+'additional_changes_rev_limit': "2",
+}
+with mock.patch('tg.confi

Re: Kallithea Setup Docs

[Mads Kiilerich]

Hi David

The Kallithea docs aim to help getting a very basic setup with the 
essentials. Something that perhaps can be used directly, but mainly 
serve as a starting point for further setup which is outside the scope 
of Kallithea. It is important to keep the configuration examples focused 
without introducing unnecessary concepts, or even worse: mixing up 
different concepts. We must assume that those who want to use advanced 
features (of uWSGI or other very configurable servers like Apache or 
Ngingx) will know how to use these or find the information elsewhere.


The uWSGI template *is* for setting up an uWSGI server. And yes, that 
uWSGI server is serving the HTTP protocol directly, not the uwsgi 
protocol. That seems like a fine setup for Kallithea, per 
https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html#can-i-use-uwsgi-s-http-capabilities-in-production 
. I assume you are asking for clarification that the template is serving 
the HTTP protocol and not the uwsgi protocol?


The first lines of the generated uWSGI section mention HTTP basics and 
configure http-socket . uWSGI is mentioned in the documentation, both 
overview and setup, but only very clearly in the context of web/http 
server. That all seems quite clear to me. Mentioning the uwsgi protocol 
doesn't seem helpful when the goal is to help people focus on the 
essentials to get something working, and enumerating things that are 
outside scope is out of scope.


We do for convenience put an [uwsgi] section inside the Kallithea .ini 
where the uwsgi binary with one of the --ini-paste options can pick it 
up. The section name is mandated by uWSGI. In a bigger setup that use 
the uwsgi protocol, there will probably be several layers of servers 
with different configuration, and you will not be using the Kallithea 
.ini file.


The --ini-paste-logged option might be a bit of an odd uWSGI feature 
that doesn't scale to bigger setups. There could *perhaps* be a point in 
giving an example or hinting towards more complex setups with a separate 
uwsgi.ini file without relying heavily on the paste configuration.


I have no doubt that uWSGI can do great things, also with the uwsgi 
protocol. As far as I can see, that can be as simple and trivial as 
using "socket" instead of "http-socket". (I can not find any 
uwsgi-socket option, and manage-script-name only seems relevant when 
using mount points.) But when using uwsgi protocol you need another 
server in front that can serve it as http. That seems like a more 
complex setup, where I doubt even less that one size fits all. I'm sure 
there are many guides and documentation that can help with that. Or is 
there something particularly relevant for Kallithea setups?


It is indeed possible to "mount" several WSGI applications inside most 
HTTP/WSGI servers (or directly in paste), but that is a more complex 
(for example because manage-script-name becomes relevant). New users 
shouldn't have to read and understand that just to get started. But that 
seems unrelated to the uwsgi protocol.


We already have some (old and possibly outdated) mentioning of setups 
with apache and ngingx etc around 
https://kallithea.readthedocs.io/en/default/setup.html#proxy-setups and 
random setup files in 
https://kallithea-scm.org/repos/kallithea/files/stable/init.d . 
Something more elaborate for uWSGI with some examples and qualified 
recommendations could fit in there.


With this context in mind, can you clarify what changes you would propose?

/Mads


On 27/11/2022 19:25, David Griffin wrote:

Hello all,

I just set up Kallithea and there's one area of the docs that could 
use clarification: emphasizing that setting up Kallithea with the 
uwsgi template sets it up to use uwsgi as an HTTP server, and not a 
uwsgi server. The name "uwsgi" is not particularly clear about this, 
because the uwsgi server application can operate multiple protocols - 
perhaps it would be better to name it as "uwsgi-http" to make it clear 
which protocol the configuration is for.


Incidentally, Kallithea appears to work great running under uwsgi as a 
uwsgi server (with the additional configuration option of 
manage-script-name = true, and setting uwsgi-socket instead of 
http-socket). This might be a better option for running behind nginx / 
lighttpd than the proxy_pass method you've got on your docs currently. 
I can write this up if it's useful.


All the best,
- David

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: About API code comments and some behavior.

[Mads Kiilerich]

Hi

Yes. Can you confirm this will solve the code problems you reported: 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/088551a24485247af328f0b8e395fdbbcd62a700 
?


/Mads


On 29/10/2022 09:28, toras wrote:

Hi

I then noticed that the enable_downloads/enable_statistics parameters 
in the create_repo API also had no effect.
It seems that no parameters are used and the repository's default 
settings are used. (I think this is probably the expected behavior.)
I think this one also needs to be removed from the API parameters and 
description.



Thanks



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: About API code comments and some behavior.

[Mads Kiilerich]

Hi

Thank you. (If you are working from a repo checkout, it would be more 
helpful if you could send a diff.) I haven't verified in detail, but all 
the changes seem plausible ;-) I have pushed them to the stable branch.


One question though:


   id : 
   result : {
 "msg": "created new gist",
-    "gist": {}
+    "gist": 
   }
   error :  null


What is gist object? That should perhaps be clarified. Or perhaps it is 
a bug that it is returned ...



- The 'parent' parameter of 'update_repo_group' does not work.
  It appears to accept the 'parent' parameter, but specifying it seems 
to have no effect. 


A quick look:  The update_repo_group API arguments seems to be handled 
by 
https://kallithea-scm.org/repos/kallithea/files/7037365a/kallithea/model/repo_group.py#L278 
. So perhaps the code in api.py should pass it as 'parent_group_id' 
instead of 'parent_group'?
(But also, 'owner' doesn't seem to be handled at all. Does owner change 
really work for you? But also, I don't think it matters much who owns a 
repo group. Admin rights does the same thing. So repo group owner should 
perhaps just be removed from api.py and documentation?)


Anyway:
I also pushed some further cleanup of the documentation in api.py and 
api.rst .That's on the *stable* branch. It would be great if they could 
converge, and we could generate api.rst from api.py .
If you want to improve documentation further, take a look at 
https://kallithea-scm.org/repos/kallithea/pull-request/325/_/api and 
propose api.py changes to make a grand unification.


/Mads


On 13/10/2022 15:00, toras wrote:

Hi.


There was an API-related topic a little while ago,so there may be 
things that are currently being changed, but I recently noticed 
something while writing a client that uses the API, so I'd like to 
report it.


- The comments in api.py have some differences from the implementation.
  I tried calling the API and attached a file that changed the part 
that is different from the result.

  (Based on commit 7037365a7bc3.)
  I don't know if this is necessary, but for your information.

- The 'parent' parameter of 'update_repo_group' does not work.
  It appears to accept the 'parent' parameter, but specifying it seems 
to have no effect.



Thanks

--
toras9000


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: repo groups API methods not documented

[Mads Kiilerich]

The failure comes from 
https://github.com/formencode/formencode/blob/1.3.1/setup.py#L14 . That 
version check doesn't handle Python 3.10 correctly. One workaround is 
thus to use Python 3.9 .


Kallithea support for 3.10 (such as using formencode 2.0) has landed in 
the stable branch, but it hasn't been released, and it had not been 
merged back to the default branch. I just fixed that.


I guess you were running at the head of the default branch. It should 
thus work for you now with Python 3.10 after "hg pull" and "hg update".


(Also, I wonder why you use "python -m pip". The virtualenv should 
contain a pip command when it is created.)


/Mads


On 9/18/22 00:12, Louis Bertrand wrote:

Hi,
The error shows up if I follow the getting_started instructions
https://kallithea.readthedocs.io/en/default/contributing.html#getting-started

The error occurs when trying to use pip to install FormEncode. See below.

I'm running Python 3.10:
 >python --version
 Python 3.10.6

I separately upgraded sphinx and that doesn't seem to be the problem.
 > sphinx-build --version
 sphinx-build 5.1.1

Here are the steps, using Windows CMD.exe (_not_ running as Administrator):
Create a venv in the kallithea folder
 > python -m venv venv
Activate the venv
 > venv\Scripts\activate.bat
Upgrade pip
 > python -m pip install -U pip
Install (possibly upgrading) setuptools
 > pip install -U setuptools

So far so good, then patch dev-requirements.txt
- Sphinx >= 1.8.0, < 3.1
+ Sphinx >= 1.8.0, < 5.2

Run the next installation:
pip install --upgrade -e . -r dev_requirements.txt python-ldap python-pam
That's when the error shows up, on FormEncode:

Collecting FormEncode<1.4,>=1.3.1
   Using cached FormEncode-1.3.1.tar.gz (197 kB)
   Preparing metadata (setup.py) ... error
   error: subprocess-exited-with-error

   × python setup.py egg_info did not run successfully.
   │ exit code: 1
   ╰─> [6 lines of output]
   Traceback (most recent call last):
 File "", line 2, in 
 File "", line 34, in 
 File 
"C:\Users\100159386\AppData\Local\Temp\pip-install-uxr2b6u8\formencode_bf6486894d3141398e0b66067c9a9a9a\setup.py",
 line 15, in 
   raise ImportError('Python version not supported')
   ImportError: Python version not supported
   [end of output]

   note: This error originates from a subprocess, and is likely not a problem 
with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.


Hope this helps
  --Louis


-Original Message-
From: Mads Kiilerich 
Sent: September 17, 2022 2:52 PM
To: Louis Bertrand ; Kallithea 

Subject: Re: repo groups API methods not documented

[EXTERNAL EMAIL - USE CAUTION]

I found that the supported Sphinx version had broken jinja2 dependencies on 
pypi - fixed by
https://kallithea-scm.org/repos/kallithea/changeset/33f1faa7fe044e6cea68bf4bf65f63e2a2dc15e9
.

With that, it works for me on linux in a venv per 
https://kallithea.readthedocs.io/en/default/contributing.html#getting-started
. I change to docs/ and run "make html". On Windows it might be easier to run 
"sphinx-build -b html -d _build/doctrees . _build/html" directly.

Exactly what error do you get, when doing what?

/Mads


On 9/17/22 18:52, Louis Bertrand wrote:

Hi,
I found the api.py and started tinkering with the doc file api.rst, but I can't build the 
HTML with Sphinx. When it imports kallithea as a module, I get an error "unsupported 
version". I'm using Python 3.10 on Windows. Should I create a venv with an earlier 
version of Python?
Thanks
   --Louis


-Original Message-
From: Mads Kiilerich 
Sent: September 17, 2022 8:55 AM
To: Louis Bertrand ; Kallithea

Subject: Re: repo groups API methods not documented

[EXTERNAL EMAIL - USE CAUTION]

Hi

That is just the documentation that is incomplete (and perhaps slightly wrong).

You can get an idea of the functionality by looking at the
implementation in
https://kallithea-scm.org/repos/kallithea/files/stable/kallithea/contr
ollers/api/api.py
.

/Mads


On 9/17/22 10:11, Louis Bertrand wrote:

Hello,
I am trying to use the API to manage a number of repo groups but I don't see 
documentation on kallithea.readthedocs.io for methods like get_repo_groups, 
create_repo_group, and so on. They seem to work. Are they undocumented because 
not stable? Or is it just a to-do?
My use case is managing a repo group for each student in a class such that they 
can all use a repo name like 'assignment1' without conflicts, and I can use the 
API to bulk clone repos to local for assessment. I need to bulk create those 
repo g

Re: repo groups API methods not documented

[Mads Kiilerich]

I found that the supported Sphinx version had broken jinja2 dependencies 
on pypi - fixed by 
https://kallithea-scm.org/repos/kallithea/changeset/33f1faa7fe044e6cea68bf4bf65f63e2a2dc15e9 
.


With that, it works for me on linux in a venv per 
https://kallithea.readthedocs.io/en/default/contributing.html#getting-started 
. I change to docs/ and run "make html". On Windows it might be easier 
to run "sphinx-build -b html -d _build/doctrees . _build/html" directly.


Exactly what error do you get, when doing what?

/Mads


On 9/17/22 18:52, Louis Bertrand wrote:

Hi,
I found the api.py and started tinkering with the doc file api.rst, but I can't build the 
HTML with Sphinx. When it imports kallithea as a module, I get an error "unsupported 
version". I'm using Python 3.10 on Windows. Should I create a venv with an earlier 
version of Python?
Thanks
  --Louis


-Original Message-
From: Mads Kiilerich 
Sent: September 17, 2022 8:55 AM
To: Louis Bertrand ; Kallithea 

Subject: Re: repo groups API methods not documented

[EXTERNAL EMAIL - USE CAUTION]

Hi

That is just the documentation that is incomplete (and perhaps slightly wrong).

You can get an idea of the functionality by looking at the implementation in 
https://kallithea-scm.org/repos/kallithea/files/stable/kallithea/controllers/api/api.py
.

/Mads


On 9/17/22 10:11, Louis Bertrand wrote:

Hello,
I am trying to use the API to manage a number of repo groups but I don't see 
documentation on kallithea.readthedocs.io for methods like get_repo_groups, 
create_repo_group, and so on. They seem to work. Are they undocumented because 
not stable? Or is it just a to-do?
My use case is managing a repo group for each student in a class such that they 
can all use a repo name like 'assignment1' without conflicts, and I can use the 
API to bulk clone repos to local for assessment. I need to bulk create those 
repo groups, and eventually delete them when the students graduate.
Thanks
   --Louis

--
Louis Bertrand, P.Eng.
Professor, Faculty of Science, Engineering and Information Technology
Durham College




This message is intended only for the named recipients. This message may 
contain information that is confidential or exempt from disclosure under 
applicable law. Any dissemination or copying of this message by anyone other 
than a named recipient is strictly prohibited. If you are not a named recipient 
or an employee or agent responsible for delivering this message to a named 
recipient, please notify us immediately, and permanently destroy this message 
and any copies you may have. Warning: Email may not be secure unless properly 
encrypted.
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general




___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: repo groups API methods not documented

[Mads Kiilerich]

Hi

That is just the documentation that is incomplete (and perhaps slightly 
wrong).


You can get an idea of the functionality by looking at the 
implementation in 
https://kallithea-scm.org/repos/kallithea/files/stable/kallithea/controllers/api/api.py 
.


/Mads


On 9/17/22 10:11, Louis Bertrand wrote:

Hello,
I am trying to use the API to manage a number of repo groups but I don't see 
documentation on kallithea.readthedocs.io for methods like get_repo_groups, 
create_repo_group, and so on. They seem to work. Are they undocumented because 
not stable? Or is it just a to-do?
My use case is managing a repo group for each student in a class such that they 
can all use a repo name like 'assignment1' without conflicts, and I can use the 
API to bulk clone repos to local for assessment. I need to bulk create those 
repo groups, and eventually delete them when the students graduate.
Thanks
  --Louis

--
Louis Bertrand, P.Eng.
Professor, Faculty of Science, Engineering and Information Technology
Durham College




This message is intended only for the named recipients. This message may 
contain information that is confidential or exempt from disclosure under 
applicable law. Any dissemination or copying of this message by anyone other 
than a named recipient is strictly prohibited. If you are not a named recipient 
or an employee or agent responsible for delivering this message to a named 
recipient, please notify us immediately, and permanently destroy this message 
and any copies you may have. Warning: Email may not be secure unless properly 
encrypted.
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Error in config value 'url_scheme_variable'

[Mads Kiilerich]

Hi

Right. Thanks for reporting.

That should be fixed with 
https://kallithea-scm.org/repos/kallithea/changeset/ed117efc9ae952bbab966a267bbd2297d31b05e2 
.


You can apply the fix manually, or install from the stable branch - 
https://kallithea.readthedocs.io/en/default/installation.html#installation-from-repository-source 
.


/Mads


On 9/13/22 17:29, toras wrote:

Hello.


I am reporting a problem I encountered when trying to deploy 
kallithhea behind a reverse proxy.

Version is v0.7.0.

Enabling the setting 'url_scheme_variable = HTTP_X_FORWARDED_PROTO' in 
the configuration file (kallithea.ini) caused an error on startup.

A portion of the stack trace output when an error occurs is shown below.


2022-09-12 19:38:41.258 ERROR [gearbox] String is not true/false: 
'http_x_forwarded_proto'

Traceback (most recent call last):
<< Omit the middle >>
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/kallithea/config/application.py", 
line 38, in 
    if any(asbool(config.get(x)) for x in ['url_scheme_variable', 
'force_https', 'use_htsts']):



Thanks

--
toras9000


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Edit user group permissions

[Mads Kiilerich]

Hi

Right. I reproduced on 
http://localhost:5000/_admin/user_groups/1/edit/perms and fixed it (and 
other similar potential problems).


See 
https://kallithea-scm.org/repos/kallithea/changeset/cc70797e70e16163c6862a42f7f045fb6782f202 
.


For now, you can upgrade to the head of the stable branch, or you can 
apply the fix manually.


/Mads



On 6/27/22 17:34, toras wrote:

Hello.


In kallithea v0.7.0.
When I press the [Add new] link on the user group permissions setting 
page, it immediately displays `User group permissions updated`.
Looking at the browser's DevTool, it appears to be making a POST 
request to the same page and getting HTTP status 302.


Is this a problem and isn't working?
Or is the functionality not yet implemented?


Thanks

--
toras9000


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH stable] hg: use correct start / stop indices when slicing revisions

[Mads Kiilerich]

Manuel,

I investigated this a bit further.

Your patch points out some problems in the code.

I missed some points in my initial reading of the patch:

- At the level of the get_changesets function, the problematic case is 
when combining start/end filters with other filters such as branch 
filtering. But I'm not aware of any way to unintentionally expose any 
actual problem.
- The mix between different "kinds" of indices is entirely inside this 
function.
- The problem is not tied to hidden changesets. The problem is seen when 
filtering by branch.
- Mercurial revision numbers is not so much the problem here. The 
problem is in whether we are using indices in self.revisions (which has 
all visible revisions) or revrange (which might be filtered by branch, 
and thus only have some of the visible revisions).


I think the patch is correct, but the code remains quite unelegant. The 
todo is kept (and moved around ... even though the old position seemed 
better). The added list comprehension for converting revrange to raw_id 
hashes seems a bit like a separate initial bugfix step, but it also 
seems to be potentially expensive ... and it can be avoided.


Alternatively, how about 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/34e66c7feb5dcde3adc32d5bacf94306d2ab99d8 
? Perhaps a bit invasive for the stable branch, but with the slow pace 
of Kallithea, it might be for the right place to put it. What's your 
thoughts on that? Do you agree with my analysis, here and in the commit 
message? And does it solve your problem?


But most important: What is the actual problem you experienced with 
this? What problem are we trying to solve?


/Mads



On 2/26/22 05:06, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1645848272 -3600
#  Sat Feb 26 05:04:32 2022 +0100
# Branch stable
# Node ID 84c94153376698aa028c03dbad1fc0dcf6f081ed
# Parent  397f73d1cdd4b39c9c17bb8d45592e866fcab88c
hg: use correct start / stop indices when slicing revisions

Previously, the indices where determined on self.revisions, which contains all
visible revisions in the repository. However, in some cases, these indices were
used on a list which had the items at different positions.

With this change, the indices are always determined on the list which is sliced
with these indices.

I didn’t fully understand the comment that I moved. If it doesn’t make sense to
move it, I’d happy to send a new patch series removing or changing the comment
before or after this change.

diff --git a/kallithea/lib/vcs/backends/hg/repository.py 
b/kallithea/lib/vcs/backends/hg/repository.py
--- a/kallithea/lib/vcs/backends/hg/repository.py
+++ b/kallithea/lib/vcs/backends/hg/repository.py
@@ -525,20 +525,9 @@
  :param branch_name:
  :param reversed: return changesets in reversed order
  """
-start_raw_id = self._get_revision(start)
-start_pos = None if start is None else 
self.revisions.index(start_raw_id)
-end_raw_id = self._get_revision(end)
-end_pos = None if end is None else self.revisions.index(end_raw_id)
-
-if start_pos is not None and end_pos is not None and start_pos > 
end_pos:
-raise RepositoryError("Start revision '%s' cannot be "
-  "after end revision '%s'" % (start, end))
-
  if branch_name and branch_name not in self.allbranches:
  msg = "Branch %r not found in %s" % (branch_name, self.name)
  raise BranchDoesNotExistError(msg)
-if end_pos is not None:
-end_pos += 1
  # filter branches
  filter_ = []
  if branch_name:
@@ -554,13 +543,22 @@
  revspec = b'all()'
  if max_revisions:
  revspec = b'limit(%s, %d)' % (revspec, max_revisions)
-revisions = mercurial.scmutil.revrange(self._repo, [revspec])
+# this is very much a hack to turn this into a list; a better 
solution
+# would be to get rid of this function entirely and use revsets
+revisions = [self._get_revision(r) for r in 
mercurial.scmutil.revrange(self._repo, [revspec])]
  else:
  revisions = self.revisions
  
-# this is very much a hack to turn this into a list; a better solution

-# would be to get rid of this function entirely and use revsets
-revs = list(revisions)[start_pos:end_pos]
+start_raw_id = self._get_revision(start)
+start_pos = None if start is None else revisions.index(start_raw_id)
+end_raw_id = self._get_revision(end)
+end_pos = None if end is None else revisions.index(end_raw_id)
+if start_pos is not None and end_pos is not None and start_pos > 
end_pos:
+raise RepositoryError("Start revision '%s' cannot be "
+  "after end revision '%s'" % (start, end))
+if end_pos is not None:
+end_pos += 1
+revs = revisions[sta

Re: [PATCH stable] vcs: use correct name of parameter “reverse” in docstring of get_changesets()

[Mads Kiilerich]

Pushed to stable - thanks.

These auto generated doc strings have no value and adds a maintenance 
burden. It would be equally good to just remove them.


/Mads



On 2/26/22 05:24, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1645849430 -3600
#  Sat Feb 26 05:23:50 2022 +0100
# Branch stable
# Node ID 2ea1e41d5d870cf3e2333ae541408129b5494120
# Parent  397f73d1cdd4b39c9c17bb8d45592e866fcab88c
vcs: use correct name of parameter “reverse” in docstring of get_changesets()

diff --git a/kallithea/lib/vcs/backends/base.py 
b/kallithea/lib/vcs/backends/base.py
--- a/kallithea/lib/vcs/backends/base.py
+++ b/kallithea/lib/vcs/backends/base.py
@@ -171,7 +171,7 @@
  :param start_date:
  :param end_date:
  :param branch_name:
-:param reversed:
+:param reverse:
  """
  raise NotImplementedError
  
diff --git a/kallithea/lib/vcs/backends/hg/repository.py b/kallithea/lib/vcs/backends/hg/repository.py

--- a/kallithea/lib/vcs/backends/hg/repository.py
+++ b/kallithea/lib/vcs/backends/hg/repository.py
@@ -523,7 +523,7 @@
  :param start_date:
  :param end_date:
  :param branch_name:
-:param reversed: return changesets in reversed order
+:param reverse: return changesets in reversed order
  """
  start_raw_id = self._get_revision(start)
  start_pos = None if start is None else 
self.revisions.index(start_raw_id)
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH stable] controllers: don’t pass start=0 to BaseRepository.get_changesets()

[Mads Kiilerich]

Pushed to stable - thanks.

/Mads


On 2/25/22 22:08, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1645822846 -3600
#  Fri Feb 25 22:00:46 2022 +0100
# Branch stable
# Node ID 397f73d1cdd4b39c9c17bb8d45592e866fcab88c
# Parent  017595560fc1e0f2acc086c63ed5f8906011d77b
controllers: don’t pass start=0 to BaseRepository.get_changesets()

MercurialRepository.get_changesets() can fail if passing start=0 if the
revision 0 is not in self.revisions. That can happen if revision 0 is not in
the visible subset of the revisions in the repository. Before Kallithea
changeset 7c43e15fb8bc7a73f17f577e59a4698589b6809d, it was working by chance
because start=0 was treated like start=None in the relevant places
(GitRepository.get_changesets still does that).

The intention of passing start=0 was seemingly to not limit the start.
Therefore passing start=None (or nothing, as it’s the default value) should be
correct.

I got the following traceback before this change:

Traceback (most recent call last):
   File "~/vcs/kallithea/kallithea/controllers/changelog.py", line 117, in index
 collection = c.db_repo_scm_instance.get_changesets(start=0, end=revision,
   File "~/vcs/kallithea/kallithea/lib/vcs/backends/hg/repository.py", line 
529, in get_changesets
 start_pos = None if start is None else self.revisions.index(start_raw_id)
ValueError: '4257f758b3eaacaebb6956d1aefc019afab956b8' is not in list

diff --git a/kallithea/controllers/changelog.py 
b/kallithea/controllers/changelog.py
--- a/kallithea/controllers/changelog.py
+++ b/kallithea/controllers/changelog.py
@@ -114,7 +114,7 @@
  webutils.flash(e, category='warning')
  raise 
HTTPFound(location=webutils.url('changelog_home', repo_name=repo_name))
  else:
-collection = c.db_repo_scm_instance.get_changesets(start=0, 
end=revision,
+collection = 
c.db_repo_scm_instance.get_changesets(end=revision,
  
branch_name=branch_name, reverse=True)
  c.total_cs = len(collection)
  
___

kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH stable] hg: use correct start / stop indices when slicing revisions

[Mads Kiilerich]

On 2/26/22 05:06, Manuel Jacob wrote:

# HG changeset patch
# User Manuel Jacob 
# Date 1645848272 -3600
#  Sat Feb 26 05:04:32 2022 +0100
# Branch stable
# Node ID 84c94153376698aa028c03dbad1fc0dcf6f081ed
# Parent  397f73d1cdd4b39c9c17bb8d45592e866fcab88c
hg: use correct start / stop indices when slicing revisions

Previously, the indices where determined on self.revisions, which contains all
visible revisions in the repository. However, in some cases, these indices were
used on a list which had the items at different positions.



I assume this is related to two different concepts:

Kallithea has this stupid non-scalable but convenient list of all 
revisions. It is used when paging, when finding next/previous changeset 
... and scanned linearly. These concepts and uses are questionable - the 
linearization of the repo is rarely really useful to users. It is better 
to walk the DAG and visit parents/children. I think it would be nice to 
get rid of self.revisions and do things differently. I think that is the 
essence of the "very much a hack" comment. But for now, we have the list 
of all visible revisions.


Mercurial also has the revision numbers, which mainly is a convenient 
implementation detail that however sometimes breaks down - for example 
when revisions are private. I don't think Kallithea these revision 
numbers except when doing low level DAG walking.


Is the real problem that Kallithea in some cases will use Mercurial 
revision numbers when it shouldn't? Or how/where do you see "items at 
different positions"? Can you say more about the problematic cases you see?


Ideally, we should add test suite coverage for your problem. But a 
manual/automated reproducible test case could also help for now.




With this change, the indices are always determined on the list which is sliced
with these indices.

I didn’t fully understand the comment that I moved. If it doesn’t make sense to
move it, I’d happy to send a new patch series removing or changing the comment
before or after this change.



The comment came from 98d235e28078 when Mercurial started encapsulating 
the internal revision list so private revisions could be hidden.


I think it still applies where you put it.


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Raw download and filename character code

[Mads Kiilerich]

Yes, now "メモ.txt" works for me too.

/Mads


On 2/8/22 14:58, toras wrote:

Hello.

I'm sorry, I should have given a concrete example.

Thank you for creating the fix code.
From the conclusion, I've confirmed that the fixed code downloaded the 
file correctly.


I tried overwriting the fixed file for kallithea installed by pip.
The repository type I tried is git, and the stored file is a name that 
contains Japanese characters.
# I don't know if this can be displayed correctly in the email, but 
the file name is "メモ.txt".



Thanks



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Raw download and filename character code

[Mads Kiilerich]

Hi

Thanks for the report.

It took a while to reproduce. Many filenames work ok. An exact example 
would have been helpful.


I reproduced the problem (or another similar one) on Linux with a 
Mercurial repository with a file created as

  python -c 'open("\U0001f4a9", "w").write("\U0001f4a9")'

I pushed a fix to the stable branch on 
https://kallithea-scm.org/repos/kallithea/changeset/8993d401575bad2d085158738b09e0bd02c7f135 
. Please let me know if that doesn't solve your problem.


/Mads


On 2/7/22 16:39, toras wrote:

Hello.

In Kallithea 0.7.0, there was a problem when raw downloading the file 
in the repository from the web interface.
It seems to be an error if the filename registered in the repository 
cannot be encoded with 'latin-1'.
It's not a big issue, as you can work around the problem by cloning 
the repository, but I'll report it here.


When an error occurs, the following will be displayed on the browser.

Internal Server Error
The server encountered an unexpected internal server error
(generated by waitress)


In addition, the following traceback is output to the console.

Traceback (most recent call last):
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/waitress/channel.py", 
line 350, in service

    task.service()
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/waitress/task.py", 
line 171, in service

    self.execute()
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/waitress/task.py", 
line 479, in execute

    self.write(chunk)
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/waitress/task.py", 
line 311, in write

    rh = self.build_response_header()
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/waitress/task.py", 
line 284, in build_response_header

    return tobytes(res)
  File 
"/home/kallithea/.local/lib/python3.8/site-packages/waitress/compat.py", 
line 69, in tobytes

    return bytes(s, "latin-1")
UnicodeEncodeError: 'latin-1' codec can't encode characters in 
position 84-85: ordinal not in range(256)



Thanks

--
toras9000


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Kallithea shows a mix of localizations

[Mads Kiilerich]

It seems like 
https://kallithea.readthedocs.io/en/0.7.0/setup.html#internationalization-i18n-support 
mostly answers the question.


The server side .ini file probably contains something like

   ## Internationalization (see setup documentation for details)
   ## By default, the languages requested by the browser are used if
   available, with English as default.
   ## Set i18n.enabled=false to disable automatic language choice.
   #i18n.enabled = true
   ## To Force a language, set i18n.enabled=false and specify the
   language in i18n.lang.
   ## Valid values are the names of subdirectories in kallithea/i18n
   with a LC_MESSAGES/kallithea.mo
   #i18n.lang = en

The server administrator can disable i18n.

But mainly, in summary: Kallithea is trying to show you the german 
localization because your web browser asks for it. You can control that 
yourself - see for example 
https://www.w3.org/International/questions/qa-lang-priorities .


/Mads


On 1/13/22 14:32, Ansis Māliņš wrote:
What I want is a 100% English UI. Why is Kallithea trying to show me 
the German localization at all? I did not find any language settings 
anywhere.


On Sun, Jan 9, 2022 at 6:57 PM Mads Kiilerich  wrote:

Hi

That is because the German translation isn't fully updated. Not
really a
bug, just work that has to be done. It is kind of tracked on
https://hosted.weblate.org/projects/kallithea/kallithea/ .

A small peek behind the scene shows for example

https://hosted.weblate.org/translate/kallithea/kallithea/de/?q=Pull+Requests+Needing+My+Review

, which currently has a very bad approximation of translation which
fortunately is ignored.

The first sections on
https://kallithea.readthedocs.io/en/stable/dev/i18n.html hints how
you
can help fix that.

Let us know if you contribute translations in weblate - then we
can make
a new release that includes it.

/Mads



On 1/9/22 13:21, Ansis Māliņš wrote:
> Hello. Kallithea is showing me a mix of English and German
> UI. Screenshot: https://u.smutty.horse/mfboqevyiav.png . Not an
> immediate problem for me, but something a bug should exist for.
If you
> need any extra info, I'm happy to provide.
>
> Kallithea version is 0.7.0
> System language is English, but I'm located in Germany.
>
> ___
> kallithea-general mailing list
> kallithea-general@sfconservancy.org
> https://lists.sfconservancy.org/mailman/listinfo/kallithea-general


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Kallithea shows a mix of localizations

[Mads Kiilerich]

Hi

That is because the German translation isn't fully updated. Not really a 
bug, just work that has to be done. It is kind of tracked on 
https://hosted.weblate.org/projects/kallithea/kallithea/ .


A small peek behind the scene shows for example 
https://hosted.weblate.org/translate/kallithea/kallithea/de/?q=Pull+Requests+Needing+My+Review 
, which currently has a very bad approximation of translation which 
fortunately is ignored.


The first sections on 
https://kallithea.readthedocs.io/en/stable/dev/i18n.html hints how you 
can help fix that.


Let us know if you contribute translations in weblate - then we can make 
a new release that includes it.


/Mads



On 1/9/22 13:21, Ansis Māliņš wrote:
Hello. Kallithea is showing me a mix of English and German 
UI. Screenshot: https://u.smutty.horse/mfboqevyiav.png . Not an 
immediate problem for me, but something a bug should exist for. If you 
need any extra info, I'm happy to provide.


Kallithea version is 0.7.0
System language is English, but I'm located in Germany.

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: migrate repository from mercurial to git

[Mads Kiilerich]

On 12/10/21 18:14, Uwe Brauer wrote:

On 12/10/21 12:29, Łukasz Michalski wrote:

Hi,

I would like to migrate a repository myrepo from mercurial to git.

I want to rename mercurial repository to myrepo_old and use orginal name for 
new git repository.

Is it enough to rename mercurial repository on disk and copy new one under 
original name? Old one can have some closed pull requests.



Yes, that should work and be safe, especially if you do the renaming
and repo creation in Kallithea. The new repo will be "different" from
the old one, even if the name is reused. Access rules and PRs etc will
be different.

I am deeply confused. Are you saying kallithea can convert a mercurial 
repository to git without any further set up?



No, that's not what I'm saying. Kallithea does not have conversion built in.

I do not see Łukasz asking about the actual conversion process. I'm sure 
he is aware of the options and trade-offs and can make the conversion 
manually. He is only asking about how he can put a new Git repo in the 
place where the Mercurial repo used to live.



It could be very interesting to develop a hosting solution that was 
completely VCS agnostic. Where the immutable main branches of each 
"repository" were exposed in "all" kinds of VCSs, so users and 
contributors could work and submit changes in any way they want. But I 
doubt anybody wants to invest enough to make that happen ;-)


It would be more simple to integrate external conversion tools so a 
click off a button in Kallithea would make a one-shot conversion to 
another repo format  with some chosen handling of the fundamental 
VCS differences.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: migrate repository from mercurial to git

[Mads Kiilerich]

On 12/10/21 12:29, Łukasz Michalski wrote:

Hi,

I would like to migrate a repository myrepo from mercurial to git.

I want to rename mercurial repository to myrepo_old and use orginal name for 
new git repository.

Is it enough to rename mercurial repository on disk and copy new one under 
original name? Old one can have some closed pull requests.



Yes, that should work and be safe, especially if you do the renaming and 
repo creation in Kallithea. The new repo will be "different" from the 
old one, even if the name is reused. Access rules and PRs etc will be 
different.


Internally, Kallithea is using numerical primary key IDs to store 
database data. The (current) repository name is only used to look up the 
ID when a user is accessing web interface or with protocol access, and 
again when finding the repository on disk. Only the access log will not 
only use the ID but laso duplicate the (current) repository name so the 
logs are useful, even if the repo is removed completely.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: httppostargs=true for Kallithea

[Mads Kiilerich]

Hi

In what way do you experience that it doesn't work with Kallithea?

Kallithea implements a custom wrapper around hgweb. Mainly to provide 
integrated hosting with correct access control. It is a basic assumption 
that reads are with GET and writes are with POST.


Without being told more about how httppostargs works / fails (and 
without spending time investigating) I would assume that it works fine 
for users with write access to repositories.


A more fine grained separation would require some refactorings ... and a 
risk of getting it wrong and introducing security issues.


If I remember correctly, the Mercurial client also has a max limit of 
how large headers it will send. But mainly: For most web servers it is 
possible to adjust the header limit to work with most repositories. That 
is apparently also possible with IIS, even if that is one of your 
constraints. Does that not work for you? Else, I would suggest using 
another web server, preferably on unix.


/Mads



On 10/11/21 15:16, Ansis Māliņš wrote:
Hello. I would really like to use Kallithea, but there's a deal 
breaker: Kallithea does not support httppostargs=true, an almost 
undocumented setting that switches Mercurial to sending heads and 
largefile hashes as POST requests instead of in headers. I've looked 
at the source code, and it seems Kallithea implements it's own version 
of hgweb. How hard would it be to add support for httppostargs? I'm 
willing to put in the work if I receive some guidance. The extent of 
my Python experience is writing Mercurial hooks and simple Discord bots.


httppostargs:
https://www.mercurial-scm.org/wiki/HgWebInIisOnWindows#I.27m_getting_an_HTTP_400:_Bad_request_error._What.27s_going_on.3F

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: 'git fetch' not working whereas 'git clone' does

[Mads Kiilerich]

(Please keep replies on the list.)

On 10/5/21 00:25, Don Zimmer wrote:


Hi,

Thanks for the reply.  I am not sure how I could reproduce it for you, 
but I will give that some thought.


Unfortunately, my server is rather special in that the repositories 
are actually residing on a mounted Windows filesystem.  One thing I 
had noticed is that the repository folders, after creation through 
Kallithea, simply contain the contents of what normally would be the 
‘.git’ hidden folder if the repository had been manually created on a 
local file system using git… *is that unexpected?*




Yes. It is using "bare repos".

Perhaps try to verify that you also see the problem if the repository is 
on a local file system. And if using a repo that just has been created 
and where you know nothing else has touched the server repo or the local 
clone.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: 'git fetch' not working whereas 'git clone' does

[Mads Kiilerich]

On 10/4/21 23:31, Don Zimmer wrote:


Hi,

I had set up an Apache server running Kallithea recently and all seems 
working well except I have just discovered that although ‘git clone’ 
works on an http URL to my server, that ‘git fetch’ fails with the 
message ‘fatal: not a git repository (or any of the parent 
directories): .git’.  Is this a known limitation?




No, we don't know about any issue like that.

The case you describe works here (when testing with gearbox serve, not 
using Apache):


$ git clone http://test_admin@localhost:5000/vcs_test_git
Cloning into 'vcs_test_git'...
Password for 'http://test_admin@localhost:5000':
remote: Enumerating objects: 3932, done.
remote: Counting objects: 100% (3932/3932), done.
remote: Compressing objects: 100% (1734/1734), done.
remote: Total 3932 (delta 2162), reused 3932 (delta 2162), pack-reused 0
Receiving objects: 100% (3932/3932), 1.13 MiB | 10.20 MiB/s, done.
Resolving deltas: 100% (2162/2162), done.

$ cd vcs_test_git/

$ git fetch
Password for 'http://test_admin@localhost:5000':
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 330 bytes | 330.00 KiB/s, done.
From http://localhost:5000/vcs_test_git
   5f2c6ee..2b209ba  master -> origin/master

$

Can you find a simple way to reproduce the problem?

/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Problem with special Characters in the URL

[Mads Kiilerich]

On 8/19/21 11:13, Rüdiger Kessel wrote:

Hi,

I was faced with an URL encoding problem which caused a 
mercurial.error.ProgrammingError in mercurial/hgweb/request.py, line 276:


"PATH_INFO does not begin with repo name"

The root cause of the encoding problem is the fact that the wsgi 
encodes the URL in 'iso-8859-1' but it is in fact utf-8 encoded. It is 
connected with the use of Python 3.


kallithea/controllers/base.py provides the function which can return 
the correct path info (get_path_info(environ)).



It could seem like Mercurial still has the problem we fixed in 
https://kallithea-scm.org/repos/kallithea/changeset/6c381371d106 .


But if the problem can be reproduced with plain 'hg serve', it should be 
fixed upstream instead of worked around in Kallithea.


/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Delete repo-group

[Mads Kiilerich]

On 7/22/21 18:13, A. Klitzing wrote:

By the way... I receive this in my log if I create a new repository
with a clone:
   Fehler: Hook changegroup.kallithea_push_action löste eine Ausnahme
aus: Environment variable KALLITHEA_EXTRAS not found
But the repository works after that.



I have noticed that too.

The hook is called late and is not fatal that the hook fails fatally.

There are some implementation details that make it tricky to fix correctly.

KALLITHEA_EXTRAS is an environment variable used to provide context for 
Mercurial and Git hooks. It is essentially a global variable that is set 
in the Kallithea worker process. (That is one of the reasons that we 
can't use multiple worker threads but have to use separate worker 
processes.)


KALLITHEA_EXTRAS carry some context information (like username and IP) 
that most of the app shouldn't care about. It is thus generally set 
early on in the request processing, so it can be used many levels and 
fuction calls later.


For repository creation there are many code paths that lead to that 
code, and it depends on data and disk state whether the repo is created 
(in which case hooks are invoked and KALLITHEA_EXTRAS is relevant), or 
if it just use an existing repo (where KALLITHEA_EXTRAS isn't used).


Some ideas for refactorings that could help:

1. Make repository creation explicit. Move it out of 
MercurialRepository/GitRepository __init__  but it currently depends 
on support from the semi-initialized Repository instance.


2. Pass the hook environment as data through all the layers and avoid 
setting the environment variable in the worker process ... or set it as 
late as possible. (For Git, it could just be set in the git subprocess. 
Mercurial hooks run in the process and might use different tricks.)


But it is also possible that this particular code path can be fixed.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [RE]Re: Kallithea 0.7.0 Running Environment Questions

[Mads Kiilerich]

On 6/17/21 7:04 AM, 김태호 wrote:


Hello Kallithea, Thank you so much.


We ended up upgradingthe EC2 instane from t2.micro to r5.xlarge.

So the specifications of the instance are as follows.

(Intel Xeon 3.1GHz Skylake-SP or Cascade Lake) vCPU = 4, RAM = 32GiB


We set the thread value to 1 when you configure the ini file, the 
download will proceed as a result, although it is slow.



The thing is the server would not be able to process anything else 
until the download is complete, so I want to increase the thread value 
more in the ini file.


I'd like to know the possible risks.



I assume you read the documentation, including 
https://kallithea.readthedocs.io/en/default/usage/performance.html#horizontal-scaling 
.



Yes, the thread value is 1 by default. And it has to stay that way.


If you have multiple simultaneous users that shouldn't slow each other 
down, you probably want multiple worker processes.



/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Kallithea 0.7.0 Running Environment Questions

[Mads Kiilerich]

Hi

Sorry, I don't understand.

Exactly what kind of pull is it you can do with Git?

Exactly what are you doing with Mercurial that doesn't work? How is it 
failing?


/Mads



On 6/23/21 8:06 AM, Park Jeonghwan wrote:


Hi,

I have one question.

My colleague and I install a Kallithea 0.7.0 and we can use the pull 
command about GIT Repo.


However, in Mercurial Repo, no function can be used.

Do I have to install additionally to use Mercurial Repo?

What am I supposed to do?

Regards,

JH Park.

*From:*Mads Kiilerich 
*Sent:* Thursday, June 17, 2021 6:07 AM
*To:* 김태호; kallithea-general@sfconservancy.org
*Cc:* 박정환
*Subject:* Re: Kallithea 0.7.0 Running Environment Questions

On 6/15/21 1:02 PM, 김태호wrote:

We are testing by installing 0.7.0 version of Kallithea in two
different environments.

One was installed on WSL2 on my Windows 10 computer, and the other
on EC2 (t2.Micro, Ubuntu20.04) on AWS.

The Kallithea git repo that I want to download is about 2.84GB.

There was no problem when I installed it on my PC to verify that
the installation process or configuration was wrong.

If I run the Kallithea in WSL2, there is no problem with the download.

The specifications of WSL2 are as follows.

CPU : Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz   1.50 GHz ( 8 core )

RAM: 16 GB

EC2 is (t2.Micro)

vCPU: 1

RAM: 1

However, if I run on an EC2 instance, the following error :

--> start

error: RPC failed; HTTP 417 curl 22 The requested URL returned
error: 417

fatal: the remote end hung up unexpectedly

--> end

Debug level log txt file is attached.

You saw this at the end of the log file:

2021-06-15 10:31:38.805 DEBUG [kallithea.config.middleware.pygrack] 
handling cmd ['git', 'upload-pack', '--stateless-rpc', 
'/var/kallithea/repos/Hicare-Smart/v2/hub-android']
2021-06-15 10:33:12.303 ERROR [kallithea.config.middleware.pygrack] 
Traceback (most recent call last):
  File 
"/home/ubuntu/kallithea/lib/python3.8/site-packages/kallithea/config/middleware/pygrack.py", 
line 160, in backend

    out = subprocessio.SubprocessIOChunker(
  File 
"/home/ubuntu/kallithea/lib/python3.8/site-packages/kallithea/lib/vcs/subprocessio.py", 
line 365, in __init__

    raise EnvironmentError("Subprocess exited due to an error: %s" % err)
OSError: Subprocess exited due to an error: b'error: pack-objects died 
of signal 9\nerror: git upload-pack: git-pack-objects died with 
error.\nfatal: git upload-pack: aborting due to possible repository 
corruption on the remote side.\n'


Kallithea is invoking 'git', and Git fails, probably because the 
server is out of memory.


You can perhaps reproduce pretty much the same problem by running this 
on the server:


    cd /var/kallithea/repos/Hicare-Smart/v2/hub-android

    git bundle create /tmp/bundle --all

On the machine where the operation works on the same repo, you can try 
to use


    /usr/bin/time -v git bundle create /tmp/bundle --all

and the line with "Maximum resident set size (kbytes)" will tell how 
much memory it is using.


While it is hard to give any advice on server size, it seems 
reasonable that the the server at least must be of similar size as the 
repo, multiplied by some factor. Next, the server size will depend on 
for example how many simultaneous operations it should handle.


I guess it would work (but be slow) if the system is configured with 
plenty of swap space. But real RAM is better.


/Mads



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Kallithea 0.7.0 Running Environment Questions

[Mads Kiilerich]

On 6/15/21 1:02 PM, 김태호 wrote:


We are testing by installing 0.7.0 version of Kallithea in two 
different environments.



One was installed on WSL2 on my Windows 10 computer, and the other on 
EC2 (t2.Micro, Ubuntu20.04) on AWS.


The Kallithea git repo that I want to download is about 2.84GB.

There was no problem when I installed it on my PC to verify that the 
installation process or configuration was wrong.



If I run the Kallithea in WSL2, there is no problem with the download.


The specifications of WSL2 are as follows.

CPU : Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz   1.50 GHz ( 8 core )

RAM: 16 GB


EC2 is (t2.Micro)

vCPU: 1

RAM: 1

However, if I run on an EC2 instance, the following error :


--> start

error: RPC failed; HTTP 417 curl 22 The requested URL returned error: 417
fatal: the remote end hung up unexpectedly
--> end

Debug level log txt file is attached.




You saw this at the end of the log file:


2021-06-15 10:31:38.805 DEBUG [kallithea.config.middleware.pygrack] 
handling cmd ['git', 'upload-pack', '--stateless-rpc', 
'/var/kallithea/repos/Hicare-Smart/v2/hub-android']
2021-06-15 10:33:12.303 ERROR [kallithea.config.middleware.pygrack] 
Traceback (most recent call last):
  File 
"/home/ubuntu/kallithea/lib/python3.8/site-packages/kallithea/config/middleware/pygrack.py", 
line 160, in backend

    out = subprocessio.SubprocessIOChunker(
  File 
"/home/ubuntu/kallithea/lib/python3.8/site-packages/kallithea/lib/vcs/subprocessio.py", 
line 365, in __init__

    raise EnvironmentError("Subprocess exited due to an error: %s" % err)
OSError: Subprocess exited due to an error: b'error: pack-objects died 
of signal 9\nerror: git upload-pack: git-pack-objects died with 
error.\nfatal: git upload-pack: aborting due to possible repository 
corruption on the remote side.\n'



Kallithea is invoking 'git', and Git fails, probably because the server 
is out of memory.



You can perhaps reproduce pretty much the same problem by running this 
on the server:


    cd /var/kallithea/repos/Hicare-Smart/v2/hub-android

    git bundle create /tmp/bundle --all


On the machine where the operation works on the same repo, you can try 
to use


    /usr/bin/time -v git bundle create /tmp/bundle --all

and the line with "Maximum resident set size (kbytes)" will tell how 
much memory it is using.



While it is hard to give any advice on server size, it seems reasonable 
that the the server at least must be of similar size as the repo, 
multiplied by some factor. Next, the server size will depend on for 
example how many simultaneous operations it should handle.



I guess it would work (but be slow) if the system is configured with 
plenty of swap space. But real RAM is better.



/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

[RE]Re: Authentication log Questions

[Mads Kiilerich]

On 6/11/21 12:07 PM, 김태호 wrote:


Hello Kallithea


Thank you for your answer !


We have solved Question1 and Question3 but the second one.



To help others with similar problems (and perhaps help us improve the 
software to avoid it), can you explain what the problem was and how you 
solved it?




So, let me elaborate on the second question.


The following attempts have been made to download kallithea-git 
repository which is about 150MB size:


1. Changed the http_server option from gearbox to waitress.

This makes it possible to download more files than the gearbox, but 
the 150MB file still fails.



*>> Command that I tired.*

ubuntu@DESKTOP-VD6IGV2:~$ git clone 
https://taeho...@scm.dev.my-company.net/my-company-smart/v2/hub-android


Cloning into 'hub-android'...

Password for 'https://taeho...@scm.dev.my-company.net:

error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504

fatal: the remote end hung up unexpectedly



I will repeat the question from 
https://lists.sfconservancy.org/pipermail/kallithea-general/2021q2/003321.html 
:



Do you get the same failure if bypassing the load balancer and 
connecting directly to the gearbox?



I also asked:


What do you get in the Kallithea logs for such requests - especially 
after setting the log levels to DEBUG?




I verified that I can serve a clone of https://github.com/git/git/ with 
gearbox, and cloning that repo locally works:


2021-06-16 15:43:38.416 INFO [kallithea.config.middleware.wrapper] 
Request from 127.0.0.1 for /baregit/git-upload-pack responded '200 OK' 
after 11.572s with 163736351 bytes




4. Changed repository setting

I changed *enable downloads *option. (checked the box) So I could 
download the zip file.


I don't think I can keep using this because all my coworkers 
using Sourcetree app. But


But I think it's going to be a clue to solve this problem.



"Downloads" is for downloading a snapshot without version control. That 
is probably not a solution to your use case.



/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Authentication log Questions

[Mads Kiilerich]

Hi

On 6/10/21 9:25 AM, 김태호 wrote:

*>> Question 1*


We are constantly getting this kind of log as following (at WARNING 
level) :


...

2021-06-10 07:04:42.989 ERROR 
[kallithea.lib.auth_modules.auth_internal] user admin had a bad password


2021-06-10 07:04:42.989 WARNI [kallithea.lib.auth_modules] User 
`admin` failed to authenticate against 
kallithea.lib.auth_modules.auth_internal


2021-06-10 07:04:48.507 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:04:48.853 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:05:18.535 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:05:18.848 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:05:48.560 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:05:48.859 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:06:18.592 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:06:18.865 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:06:39.255 WARNI [kallithea.lib.auth_modules] User 
`member1` failed to authenticate against 
kallithea.lib.auth_modules.auth_crowd


2021-06-10 07:06:45.635 WARNI [kallithea.lib.auth_modules] User 
`member1` failed to authenticate against 
kallithea.lib.auth_modules.auth_crowd


2021-06-10 07:06:48.719 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:06:48.884 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


[2021-06-10 07:07:10 +] [5664] [INFO] Handling signal: winch

2021-06-10 07:07:18.640 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:07:18.911 WARNI [kallithea.lib.auth] user is anonymous 
and NOT authenticated with regular auth @ HomeController:index


2021-06-10 07:07:22.535 WARNI [kallithea.lib.auth_modules] User 
`member1` failed to authenticate against 
kallithea.lib.auth_modules.auth_crowd


2021-06-10 07:07:28.507 WARNI [kallithea.lib.auth_modules] User 
`member1` failed to authenticate against 
kallithea.lib.auth_modules.auth_crowd


...


I wonder why the log prints that the authentication failed even if 
login is successful.




It is very possible that some of the logging statements in the code are 
too verbose. The code path might seem warning-ish when looking at the 
particular function, but perhaps less relevant when used in production. 
We can fix that.



For example, I noticed that one of the messages seems too verbose and 
not helpful, so I will propose something like 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/2b37222f8d71 
for next release.



Even if not a Python programmer, it should be possible for you to find 
the log statements in the code and get an idea under what circumstances 
it is emitted. That can provide a good argument why the logging should 
be different.



For the log entries you show, the timestamps are so different that it 
must be for different requests. INFO logging for [logger_kallithea] will 
give more context ... but also make the logging much more verbose and 
impossible to read if there are multiple simultaneous requests.



Can you be more precise about which "wrong" log statements come from 
*one* successful login?




*>> Question 2*

*
*

We changed http_server to waitress to download a large size file from 
my kallithea-git.


Also I put larger value for channel_timeout 
and max_request_body_size(channel_timeout = 180, max_request_body_size 
= 322122547200) in my .ini file.


But I still haven't received the same size file as the next attached 
picture.



*But I can still download small size files from Kallithea. *

image.16233092690410.png



(Is this related to the other question asked some days ago? If so: It is 
much better to follow up in that thread so we have all the relevant 
information in one place, without mixing things up. And make sure to 
answer all questions if you want help.)



Exactly what problem do you see? Can you reproduce it in a test setup? 
Can you describe how you think we can reproduce the problem?




*>> Question 3*

As a guest, whenever I push to git repositoy, the following error 
message occurs.



*ERROR MESSAGE 1:*

POST git-receive-pack (642 bytes)
remote: Traceback (most recent call last):
remote:   File "hooks/pre-receive", line 16, in 
remote: from kallithea.lib.hooks import handle_git_pre_receive as _handler 


remot

Re: Kallithea git big file pull/push timeout error question

[Mads Kiilerich]

I think Kallithea is using gearbox in a pretty standard way. You can 
tweak your .ini to change the configuration.


What do you get in the Kallithea logs for such requests - especially 
after setting the log levels to DEBUG?



On 6/7/21 8:12 AM, 김태호 wrote:


Hello Kallithea

We are currnetly using Kallithea on AWS EC2 instance.


The sturcture of it is follow as:

AWS Load Balancer -> EC2 (Kallithea)


The ini file has been made by the CLI of :

kallithea-cli config-create my.ini database_engine=postgres 
http_server=gearbox



Whenever my team try to download this big file(approx 2GB), following 
error comes out:


RPC failed; HTTP 504 curl 22 The requested URL returned error: 504



504 sounds like an error from the proxy server.

Do you get the same failure if bypassing the load balancer and 
connecting directly to the gearbox? If not, the amount of data just 
takes more time than the load balancer allows, and you will have to 
change the load balancer configuration.


Related to this, I wonder if there is the file that can change the 
settings for timeout for gearbox or something else.




I don't know what to say about that. Kallithea is using gearbox in a 
pretty standard way, with minimal configuration in the .ini . You should 
also be able to set any timeout settings you might have found in the 
gearbox documentation.



/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Hello again Kallithea I got a question with Crowd Authentication Plugin

[Mads Kiilerich]

Hi

I have no magic solution to "doesn't work". We will need more info.

Exactly how did you configure crowd auth in Kallithea? On which address 
is your crowd instance running?


How do you experience it doesn't work?

What do you get in the server log when it doesn't work? Especially with 
[logger_kallithea] level=DEBUG .


/Mads


On 6/4/21 7:46 AM, 김태호 wrote:

Hello :)

I am setting up the Crowd Authentication Plugin.

Crowd server is on another EC2 instance, and we obviously checked 
everything. We have tried the Domain name and IP of the Crowd page too.


But this does not work

I wonder what else we can do on Kallithea Auth settings?
Or should I talk to the Atlassian team?

Regards

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Hello Kallithea :) I have a question

[Mads Kiilerich]

Hi

No, we haven't made any recommendations. It depends so much on the size 
of repositories and the amount of usage and traffic. For private use, it 
is probably fine if it can serve one page per second, so everything is 
fine as long as it doesn't start swapping. Memory will probably be the 
bottleneck, so perhaps aim for 1 or 2 GB, and change if it turns out to 
be enough or too little for your use.


/Mads


On 5/25/21 9:32 AM, 김태호 wrote:

Hello Kallithea :) I have a question
I am planning a toy project with a bunch of groups of people in 
my school.
So I want to create a server on AWS to have Kallithea, and I wonder if 
there is a minimum or recommended hardware specification.


Regards :)

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Issue reading configuration from .hg/hgrc inside project

[Mads Kiilerich]

On 4/21/21 1:50 PM, Mads Kiilerich wrote:

On 4/21/21 2:09 AM, Felipe Cardoso Resende wrote:

Hi,

I'm experimenting with kallithea but I'm having trouble making it
read from the .hg/hgrc inside the project. Looking through
the history, I found the revision 802fdeefc8cc linking to
Issue #246 [1] that changes how the configuration is read.

I looked into mercurial.ui.ui.readconfig and it expects a filename 
instead
of the repo_path. There is a Note there that seems to imply that 
.hg/hgrc will

be read elsewhere, but I'm not that familiar with mercurial.

Does anyone knows if its really a bug or if I'm missing something?



Yes, it could seem like I made a mistake in 
https://kallithea-scm.org/repos/kallithea/changeset/f83326e2e66c .



A fix has been pushed to the stable branch: 
https://kallithea-scm.org/repos/kallithea/changeset/c9834271cd066d4f734b0f7c56e94866057a789f


Some additional cleanup in this area has been pushed to the default 
branch and will probably soon be released in 0.7 .


Thanks for the report.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Changing .ssh/authorized_keys script

[Mads Kiilerich]

On 5/19/21 4:21 PM, Louis Bertrand wrote:

Hello,

When I upgraded Kallithea, I somehow changed the venv directory from 
~/kallithea-venv to kallithea/venv. In other words, instead of a venv directory 
beside the kallithea directory, it's now under the directory.

Everything is fine except for ssh access because the cli script has moved. I 
updated the path

kallithea_cli_path = 
/srv/web/sites/hg.set.durhamcollege.org/kallithea/venv/bin/kallithea-cli
However the .ssh/authorized_keys file needs to be updated.


There is a way to update authorized_keys. See the last section in 
https://kallithea.readthedocs.io/en/default/setup.html#using-kallithea-with-ssh 
: The ``authorized_keys`` file can be updated manually with 
``kallithea-cli ssh-update-authorized-keys -c my.ini``.





Is there a way to edit the file without invalidating it?



(I'm not sure what is meant with "edit without invalidating" - is it 
"make it rewrite the file even though nothing changed that could trigger 
it"?)


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Issue reading configuration from .hg/hgrc inside project

[Mads Kiilerich]

On 4/21/21 2:09 AM, Felipe Cardoso Resende wrote:

Hi,

I'm experimenting with kallithea but I'm having trouble making it
read from the .hg/hgrc inside the project. Looking through
the history, I found the revision 802fdeefc8cc linking to
Issue #246 [1] that changes how the configuration is read.

I looked into mercurial.ui.ui.readconfig and it expects a filename instead
of the repo_path. There is a Note there that seems to imply that .hg/hgrc will
be read elsewhere, but I'm not that familiar with mercurial.

Does anyone knows if its really a bug or if I'm missing something?



Yes, it could seem like I made a mistake in 
https://kallithea-scm.org/repos/kallithea/changeset/f83326e2e66c .


What hgrc settings would you like to use with Kallithea? What is your 
test case that shows the problem?



/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Kallithea crashes when "IP address" headers have hostnames

[Mads Kiilerich]

On 4/13/21 4:29 PM, Brett Smith wrote:


Hi Kallithea team,

I got this crash report I thought I should pass on. The short version: 
some IP address/Internet mapping service visited us, and provided a 
full DNS hostname in the various IP address headers. The code crashes 
because it assumes any string in these headers /must/ be an IP 
address, without checking.


I'm personally not particularly worried about this bug, since this 
obviously isn't a "real" visitor and I'm sure Kallithea isn't the only 
software out there making this assumption. But I also know how 
sometimes one bug can lead to another, so I wanted to let you know at 
least. 23.253.224.235 is the IPv4 address of our Kallithea server, so 
the way it appears in the header values here is part of how this 
mapping project works. Let me know if there's any other information I 
can provide that's helpful.




Thanks for the report. We will improve the handling of invalid client 
addresses.



But I'm surprised your webserver (waitress?) according to the 
environment dump apparently didn't set REMOTE_ADDR from the actual TCP 
connection in the environment.


The CGI spec (rfc 3875) says: "The REMOTE_ADDR variable MUST be set to 
the network address of the client sending the request to the server".


The WSGI spec (pep 333) says: "A server or gateway **should** attempt to 
provide as many other CGI variables as are applicable".


REMOTE_ADDR might be less relevant if it just points at a front-end 
server, but I would expect it to be set anyway.


/Mads



On 4/12/21 11:33 AM, Conservancy Kallithea wrote:

TRACEBACK:
Traceback (most recent call last):
   File "/usr/local/src/kallithea/lib/python3.7/site-packages/tg/wsgiapp.py", 
line 82, in __call__
 response = self.wrapped_dispatch(controller, environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/appwrappers/errorpage.py",
 line 104, in __call__
 resp = self.next_handler(controller, environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/appwrappers/caching.py",
 line 54, in __call__
 return self.next_handler(controller, environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/appwrappers/session.py",
 line 71, in __call__
 response = self.next_handler(controller, environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/appwrappers/i18n.py", 
line 71, in __call__
 return self.next_handler(controller, environ, context)
   File "/usr/local/src/kallithea/lib/python3.7/site-packages/tg/wsgiapp.py", 
line 243, in _dispatch
 return controller(environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/base.py", 
line 511, in __call__
 ip_addr=ip_addr,
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/base.py", 
line 458, in _determine_auth_user
 authuser = AuthUser.make(dbuser=default_user, ip_addr=ip_addr)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/auth.py", 
line 391, in make
 if not check_ip_access(source_ip=ip_addr, allowed_ips=allowed_ips):
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/auth.py", 
line 806, in check_ip_access
 if ipaddr.IPAddress(source_ip) in ipaddr.IPNetwork(ip):
   File "/usr/local/src/kallithea/lib/python3.7/site-packages/ipaddr.py", line 
83, in IPAddress
 address)
ValueError: '23-253-224-235-xrip.DOMAIN' does not appear to be an IPv4 or IPv6 
address


ENVIRON:
CONTENT_LENGTH: '0'
HTTP_ACCEPT: '*/*'
HTTP_ACCEPT_ENCODING: 'gzip'
HTTP_CLIENT_IP: '23-253-224-235-cip.DOMAIN'
HTTP_CONNECTION: 'Keep-Alive'
HTTP_CONTACT: 'root@23-253-224-235-con.DOMAIN'
HTTP_FROM: 'root@23-253-224-235-from.DOMAIN'
HTTP_HOST: '23.253.224.235'
HTTP_REFERER: 'https://23-253-224-235-ref.DOMAIN/ref'
HTTP_TRUE_CLIENT_IP: '23-253-224-235-tcip.DOMAIN'
HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64; rv:73.0) 
Gecko/20100101 Firefox/73.0root@user-agent.DOMAIN'
HTTP_X_CLIENT_IP: '23-253-224-235-xcip.DOMAIN'
HTTP_X_FORWARDED_SERVER: 'k.sfconservancy.org'
HTTP_X_ORIGINATING_IP: '23-253-224-235-xoip.DOMAIN'
HTTP_X_REAL_IP: '23-253-224-235-xrip.DOMAIN'
PATH_INFO: '/error/document'
QUERY_STRING: ''
REQUEST_METHOD: 'GET'
SCRIPT_NAME: ''
SERVER_PROTOCOL: 'HTTP/1.1'
SERVER_SOFTWARE: 'waitress'


WSGI:
backlash.exc_environ: {'REQUEST_METHOD': 'GET', 'SERVER_SOFTWARE': 'waitress', 'SERVER_PROTOCOL': 'HTTP/1.1', 'SCRIPT_NAME': 
'', 'PATH_INFO': '/', 'QUERY_STRING': '', 'wsgi.url_scheme': 'https', 'wsgi.version': (1, 0), 'wsgi.errors': <_io.TextIOWrapper 
name='' mode='w' encoding='UTF-8'>, 'wsgi.multithread': True, 'wsgi.multiprocess': False, 'wsgi.run_once': False, 
'wsgi.input': <_io.BytesIO object at 0x7f60d84b69e8>, 'wsgi.file_wrapper': , 'wsgi.input_terminated': Tru

Re: IndexError crash during pagination on 0.6.3

[Mads Kiilerich]

Hi

Thanks for the report and the test case.

Yes, that comment is wrong. It didn't consider that if the page number 
is too high, the resulting page will be empty even though the first page 
isn't.


I propose fixing it with 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/0ff78683c8730a454b9df805afe3766eebfa5a62


/Mads


On 2/1/21 4:45 PM, Brett Smith wrote:


Hi all,

I got this crash report from the server at k.sfconservancy.org over 
the weekend, and the comment in the last line jumped out at me. It 
sounds like there's a situation where the code assumes a list can't be 
empty, but in fact it can be. The full details are below, except I've 
removed bits that just reveal too much about our server setup or the 
visitor.


You're welcome to browse around or clone the repository if it helps 
you debug the problem; it's all public.



TRACEBACK:
Traceback (most recent call last):
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/appwrappers/session.py",
 line 71, in __call__
 response = self.next_handler(controller, environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/appwrappers/i18n.py", 
line 71, in __call__
 return self.next_handler(controller, environ, context)
   File "/usr/local/src/kallithea/lib/python3.7/site-packages/tg/wsgiapp.py", 
line 243, in _dispatch
 return controller(environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/base.py", 
line 536, in __call__
 return super(BaseController, self).__call__(environ, context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/controllers/dispatcher.py",
 line 118, in __call__
 response = self._perform_call(context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/controllers/dispatcher.py",
 line 107, in _perform_call
 r = self._call(action, params, remainder=remainder, context=context)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/controllers/decoratedcontroller.py",
 line 129, in _call
 output = controller_caller(context_config, bound_controller_callable, 
remainder, params)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/controllers/decoration.py",
 line 21, in _decorated_controller_caller
 return application_controller_caller(tg_config, controller, remainder, 
params)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/tg/configurator/components/dispatch.py",
 line 114, in _call_controller
 return controller(*remainder, **params)
   File "", line 2, in index
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/auth.py", 
line 608, in __wrapper
 return func(*fargs, **fkwargs)
   File "", line 2, in index
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/lib/auth.py", 
line 657, in __wrapper
 return func(*fargs, **fkwargs)
   File 
"/usr/local/src/kallithea/lib/python3.7/site-packages/kallithea/controllers/changelog.py",
 line 148, in index
 c.first_revision = c.cs_pagination[0] # pagination is never empty here!
IndexError: list index out of range
ENVIRON:
HTTP_ACCEPT: '*/*'
HTTP_ACCEPT_CHARSET: 'utf-8;q=0.7,iso-8859-1;q=0.2,*;q=0.1'
HTTP_CONNECTION: 'Keep-Alive'
HTTP_HOST: 'k.sfconservancy.org'
HTTP_USER_AGENT: 'Mozilla/5.0 (compatible; 
DotBot/1.1;http://www.opensiteexplorer.org/dotbot,h...@moz.com)
PATH_INFO: 
'/website/changelog/49c5bca7430b19dd09e0e12d2b908b86d278d325/www/conservancy/static/supporter/index.html'
QUERY_STRING: 'page=2'
REQUEST_METHOD: 'GET'
SCRIPT_NAME: ''
SERVER_PROTOCOL: 'HTTP/1.1'
SERVER_SOFTWARE: 'waitress'
WSGI:
beaker.session: {'_domain': None, '_path': '/', '_accessed_time': 
1612166635.5476, '_creation_time': 1612166635.5476}
pylons.routes_dict: {'repo_name': 'website', 'revision': 
'49c5bca7430b19dd09e0e12d2b908b86d278d325', 'f_path': 
'www/conservancy/static/supporter/index.html', 'action': 'index', 'controller': 
'changelog'}
tg.routes_dict: {'repo_name': 'website', 'revision': 
'49c5bca7430b19dd09e0e12d2b908b86d278d325', 'f_path': 
'www/conservancy/static/supporter/index.html', 'action': 'index', 'controller': 
'changelog'}
webob._parsed_query_vars: (GET([('page', '2')]), 'page=2')
webob.adhoc_attrs: {'authuser': , 'ip_addr': '[redacted]', 
'needs_csrf_check': False, 'routes_local': {'mapper': , 'host': 'k.sfconservancy.org', 'protocol': 'https', 'redirect': 
}}
wsgi.input_terminated: True
wsgi.multiprocess: False
wsgi.multithread: True
wsgi.run_once: False
wsgi.url_scheme: 'https'
wsgi.version: (1, 0)
wsgiorg.routing_args: (, {'repo_name': 'website', 'revision': 
'49c5bca7430b19dd09e0e12d2b908b86d278d325', 'f_path': 
'www/conservancy/static/supporter/index.html', 'action': 'index', 'controller': 
'changelog'})
REQUEST:
https://k.sfconservancy.or

Re: Behavior of create_repo API as a general user

[Mads Kiilerich]

This has now been fixed in the stable branch and will be included in the 
next release - no matter if it will be 0.6.4 or 0.7 .


Thanks for the report.

/Mads


On 1/3/21 2:38 AM, toras wrote:

Hi

Thank you for your answer.

I understand that it is a process of changing the interpretation of 
this setting value.

I'm looking forward to future versions including behavior fixes.

Thanks


toras9000

On 2021/01/03 7:49, Mads Kiilerich wrote:

Hi

You are right. Kallithea has some bugs around API permission 
handling. It is not

using the "create top-level repositories" permissions correctly.

This problem is related to the
"This will also give all users API access to create repositories 
everywhere.

That might change in future versions."
note, even though you see the opposite problem.

This behaviour is kind of intentional -
https://kallithea-scm.org/repos/kallithea/changeset/6620542597d3 - 
and with some

awareness in the test suite -
https://kallithea-scm.org/repos/kallithea-incoming/changeset/975f5769be08 
...

but doesn't match what hg.create.repositoryactually means:
https://kallithea-scm.org/repos/kallithea/changeset/8aad6a324739#kallitheamodeldbpy_n1676 



I propose
https://kallithea-scm.org/repos/kallithea/pull-request/303/_/api_permission_check 
to

fix this.

/Mads




___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] git: move non-kallithea hooks and execute other hooks

[Mads Kiilerich]

On 1/11/21 10:16 PM, Lance Edgar wrote:


On 1/11/21 7:53 AM, Mads Kiilerich wrote:
As next step, I propose the changes in 
https://kallithea-scm.org/repos/kallithea/pull-request/306/_/hooks .


Some of it took a slightly different direction than Tim proposed. And 
cleaning up many other related things. Please let us know if you 
disagree with any of it.



The only thing I spotted in the above PR was this bit where you would 
remove any existing pre-receive hook.  Seems to me that the admin may 
well need to install pre-receive for their own needs and you should 
never delete it even when "force overwrite" is requested?  Or maybe I 
read it wrong.


https://kallithea-scm.org/repos/kallithea-incoming/changeset/e7acbdb8fd8ce7d520ce216b5baeb42a72dc7fdd#kallitheamodelscmpy_n717 




Right, thanks. I propose 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/9522b915c913#kallitheamodelscmpy_n674 
instead.



Finally, with these things out of the way, the core problem of 
custom/multiple Git hooks is more clear.


The way Kallithea use Git, I see no alternative to Kallithea 
installing the post-receive hook to be able to detect what has been 
pushed. (But it is possible we could come of with another scheme, and 
thus completely avoid this problem?)



Agreed, I see no reason not to use git hooks for Kallithea's own needs.



Well ... there is at least one good reason: it evidently conflicts with 
admins who want other kinds of hooks ;-)



I don't like the idea of Kallithea having its own custom way of 
allowing admins to install hooks. If admins want to install extra 
hooks, they should do it in a "standard" way. Tim mentioned a couple 
of examples of how it could be done, but nothing that was ready for 
use, covered relevant concerns such as having a scheme that didn't 
pick up backup files or non-executable files, with a convincing cross 
platform story (covering Windows), or approximated a "standard".


I think the best (and preferred) way forward for having custom and 
multiple hooks for Git would be to have a "standard" dispatcher that 
can be installed in hooks/HOOKNAME . That hook should be a separate 
project, independent of Kallithea, and with a fair chance of being 
used widely, also when not using Kallithea. Ideally, it should be 
upstreamed in Git.


The scheme could be that it runs all executable 
hooks/HOOKNAME.d/*.hook (or some other scheme, such as 
hooks/HOOKNAME.*.hook). It should have some meaningful handling of 
stdin, stdout, exit code, and how they should be shared/chained when 
multiple hooks run and might fail. And it should support both Linux, 
Mac, and Windows (where there might be cmd/exe/whatever extensions 
and no exe flag).


When there is a good 3rd party solution to the problem, I will 
implement support for it in Kallithea: detect if a hook from that 
project is installed in hooks/HOOKNAME, and then write the Kallithea 
hook to something like hooks/HOOKNAME.d/kallithea.hook instead.


I suggest you team up to create that project. We will support you, 
but Kallithea would not be a good owner of it.


Does that make sense?



All that does make sense.  However I would like to propose yet another 
way of looking at it.


We have been discussing "multiple post-receive hooks" but in reality 
there need only ever be 2:  Kallithea's own "internal" hook script (in 
use today), and "some other" script as defined by admin.


Git after all will only support one script out of the box, so any 
admin who needs "multiple" things to happen in a hook, normally must 
write one script which contains several steps.  So then "all extra" 
steps needed beyond Kallithea could be represented as a single 
script.  If the admin's needs are complex then maybe that script for 
them must have the "meaningful handling of stdin" etc. which you 
mentioned.  (Perhaps with auto-discovery, like example at 
https://serverfault.com/a/909154 .)  But there would be no true need 
for the "dispatcher" to have those smarts - it need only a) run the 
Kallithea hook, and then b) run the "other" hook script if present.  
(I guess, it *would* still need to correctly pass stdin at least.)



I do not like to see it as at most two hooks, Kallithea's and the 
admins. That will force admins to solve the same problem as we have to: 
Figure out a way to dispatch to multiple hooks. That is an artificial 
and egoistic way to reduce a generic N problem to a "me vs everybody 
else" 2 problem.



I agree that a smarter dispatcher might be a nice gift to the Git 
world, beyond Kallithea.  But I think it is not strictly necessary here.


I was curious about what GitLab did for this.  IIUC their Server Hooks 
doc (https://docs.gitlab.com/ee/administration/server_hooks.html) 
basically reduces the problem in the same way.  They allow the admin 
to create a `custom_h

Re: [PATCH] git: move non-kallithea hooks and execute other hooks

[Mads Kiilerich]

https://kallithea-scm.org/repos/kallithea/changeset/Thanks for the 
reminder of this discussion. Sorry for dropping it.



As spin-off from this discussion and Tim Ooms' patch proposal, we landed 
some changes in the default branch, heading for 0.7 :


https://kallithea-scm.org/repos/kallithea/changeset/6a90b1ebea2c
hooks will now always be written atomically and will never follow symlinks

https://kallithea-scm.org/repos/kallithea/changeset/1089fac66e81
detect hooks that are symlinks as having been changed by the sysadmin, 
and they will thus (by default) not be overwritten


https://kallithea-scm.org/repos/kallithea/changeset/7b809e4a1ea5
use current umask when writing hooks file

https://kallithea-scm.org/repos/kallithea/changeset/5de682ee0b06
introduce cli repo-scan options --install-git-hooks and 
--overwrite-git-hooks


https://kallithea-scm.org/repos/kallithea/changeset/92894fccc615
recommend always running repo-scan to reinstall Git hooks after upgrading

That's a start.



As next step, I propose the changes in 
https://kallithea-scm.org/repos/kallithea/pull-request/306/_/hooks .


Some of it took a slightly different direction than Tim proposed. And 
cleaning up many other related things. Please let us know if you 
disagree with any of it.




Finally, with these things out of the way, the core problem of 
custom/multiple Git hooks is more clear.


The way Kallithea use Git, I see no alternative to Kallithea installing 
the post-receive hook to be able to detect what has been pushed. (But it 
is possible we could come of with another scheme, and thus completely 
avoid this problem?)


I don't like the idea of Kallithea having its own custom way of allowing 
admins to install hooks. If admins want to install extra hooks, they 
should do it in a "standard" way. Tim mentioned a couple of examples of 
how it could be done, but nothing that was ready for use, covered 
relevant concerns such as having a scheme that didn't pick up backup 
files or non-executable files, with a convincing cross platform story 
(covering Windows), or approximated a "standard".


I think the best (and preferred) way forward for having custom and 
multiple hooks for Git would be to have a "standard" dispatcher that can 
be installed in hooks/HOOKNAME . That hook should be a separate project, 
independent of Kallithea, and with a fair chance of being used widely, 
also when not using Kallithea. Ideally, it should be upstreamed in Git.


The scheme could be that it runs all executable hooks/HOOKNAME.d/*.hook 
(or some other scheme, such as hooks/HOOKNAME.*.hook). It should have 
some meaningful handling of stdin, stdout, exit code, and how they 
should be shared/chained when multiple hooks run and might fail. And it 
should support both Linux, Mac, and Windows (where there might be 
cmd/exe/whatever extensions and no exe flag).


When there is a good 3rd party solution to the problem, I will implement 
support for it in Kallithea: detect if a hook from that project is 
installed in hooks/HOOKNAME, and then write the Kallithea hook to 
something like hooks/HOOKNAME.d/kallithea.hook instead.


I suggest you team up to create that project. We will support you, but 
Kallithea would not be a good owner of it.


Does that make sense?


/Mads



On 1/8/21 11:50 PM, Lance Edgar wrote:
I'm fairly new to Kallithea but also encountered this problem with my 
setup.  I'm not clear where things are at with this discussion now, 
but it seems unresolved so wanted to share my perspective.


I need the same thing others have mentioned:  to effectively have 
multiple post-receive hooks for certain git repos.  Possible solutions 
appear to be:


A) Kallithea hook gets built-in support for running additional hooks, 
discovered by globbing


B) admin must create a Kallithea extension which somehow does that 
same thing


C) admin installs custom hook script which can run the Kallithea hook, 
plus additional (per glob)


Thus far I've gone with option C, because option A doesn't exist and 
option B seems wrong to me.  The only thing I don't really like about 
C is that Kallithea will no longer update its own hook script.  If I'm 
being honest I think option A should exist and if so I'd use it.



On 11/4/20 3:14 AM, Tim Ooms wrote:

On 24/10/2020 13:05, Mads Kiilerich wrote:

On 10/23/20 9:30 AM, Tim Ooms wrote:
It might have rough edges to bypass Kallithea when accessing 
Kallithea-managed repos. It is perhaps better to go with the 
assumption that Kallithea fully owns the repositories it manage.


I tried to not bypass Kallithea by letting it execute the other 
hooks. I know the mercurial hooks are more integrated, but as the 
git hooks don't work the same way I think this is the next best 
thing. Although there may be better ways, I'm not sure.



If you only access the repos through Kallithea, I think it would be 
better to run your hooks as Kallithe

Re: Behavior of create_repo API as a general user

[Mads Kiilerich]

Hi

You are right. Kallithea has some bugs around API permission handling. 
It is not using the "create top-level repositories" permissions correctly.


This problem is related to the
"This will also give all users API access to create repositories 
everywhere. That might change in future versions."

note, even though you see the opposite problem.

This behaviour is kind of intentional - 
https://kallithea-scm.org/repos/kallithea/changeset/6620542597d3 - and 
with some awareness in the test suite - 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/975f5769be08 
... but doesn't match what hg.create.repositoryactually means: 
https://kallithea-scm.org/repos/kallithea/changeset/8aad6a324739#kallitheamodeldbpy_n1676


I propose 
https://kallithea-scm.org/repos/kallithea/pull-request/303/_/api_permission_check 
to fix this.


/Mads


On 1/2/21 7:20 PM, toras wrote:

Hi

I have doubts about the behavior of 'create_repo' in Kallithea's API, 
so I will post it.

The version of kallithea I'm using is 0.6.3.

A 'create_repo' request to a repository group for which the account 
has write permissions also appears to fail if top-level repository 
creation is disabled.
The same request succeeds when I enable the create top-level 
repository setting.
Regardless of top-level settings, I can use that account to create 
repositories from the web into repository groups.


I didn't understand if the explanation of 'Note' on the setting screen 
means "Failed even if I have write permission".


For the time being, the situation I tried is described below.

The request was made like this.
```
curl http://localhost:5000/_admin/api -X POST -H 
'content-type:text/plain' --data-binary 
'{"id":1,"api_key":"0ae8322ce787f08771c6b3570765318fb0360ad6","method":"create_repo","args":{"repo_name":"grp/test", 
"repo_type":"git"}}'

```

The response in case of failure is like this.
```
{"id": 1, "result": null, "error": "Internal server error"}
```

The console output of kallithea at that time looks like the following.
```
2021-01-02 17:25:23.087 DEBUG [JSONRPC] Trying to find JSON-RPC 
method: create_repo
2021-01-02 17:25:23.087 INFO  [JSONRPC] IP: 127.0.0.1 Request to 
/_admin/api time: 0.012s
2021-01-02 17:25:23.127 ERROR [JSONRPC] Encountered unhandled 
exception: Traceback (most recent call last):
  File 
"/home/kallithea/.local/lib/python3.6/site-packages/kallithea/controllers/api/__init__.py", 
line 225, in _rpc_call

    raw_response = getattr(self, action)(**rpc_args)
  File "", line 2, in create_repo
  File 
"/home/kallithea/.local/lib/python3.6/site-packages/kallithea/lib/auth.py", 
line 664, in __wrapper

    raise HTTPForbidden()
webob.exc.HTTPForbidden: Access was denied to this resource.
```

# I rely on translation tools. I'm sorry if there is a strange sentence.


Thanks


toras9000

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Changelog table crashes if it tries to render a commit with an empty message

[Mads Kiilerich]

We landed the fix as 
https://kallithea-scm.org/repos/kallithea/changeset/f950740985f47ea09ef3361fa56b3d7581429156


Thanks,
Mads


On 12/15/20 4:10 PM, Brett Smith wrote:

On 12/14/20 5:15 PM, Mads Kiilerich wrote:
Ouch, yeah, .splitlines() is confusingly different from .split('\n') 
. Thanks for the patch.


Would it perhaps be better to just use .split instead of .splitlines()?


Your call but on balance I do think splitlines is better. It splits on 
all kinds of Unicode line separators 
<https://docs.python.org/3/library/stdtypes.html#str.splitlines>, not 
just \n, and I think that's good when you're dealing with very 
arbitrary user text like this.


But since the commit message text is shown as a link, would it 
perhaps be even better to avoid the void and show a text like "(No 
commit message)"?


I think that would be nice. It wasn't in my patch because I wasn't 
sure what all was involved with adding a new message string to the 
code, but if you can make it happen I think it would be worth it.


In general I didn't think my patch was thorough enough to apply 
directly to the code. I sent it just to communicate exactly where I 
found the bug.


Thanks,

--
Brett Smith



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Changelog table crashes if it tries to render a commit with an empty message

[Mads Kiilerich]

Ouch, yeah, .splitlines() is confusingly different from .split('\n') . 
Thanks for the patch.


Would it perhaps be better to just use .split instead of .splitlines()?

But since the commit message text is shown as a link, would it perhaps 
be even better to avoid the void and show a text like "(No commit message)"?


And if it is possible to have an empty commit message, should we perhaps 
also have nice handling of the case where the first line is empty?


So something like

--- kallithea/templates/changelog/changelog_table.html
+++ kallithea/templates/changelog/changelog_table.html
@@ -55,7 +55,7 @@
 
   title="${h.fmt_date(cs.date)}">${h.age(cs.date,True)}

 
-    <% message_lines = cs.message.splitlines() %>
+    <% message_lines = cs.message.strip().splitlines() or [_("(No 
commit message)")] %>

 %if len(message_lines) > 1:
 
   

Also, I can see we also have the same problem in pullrequest_show.html ...

/Mads


On 12/14/20 10:04 PM, Brett Smith wrote:


Hi,

I recently upgraded a Kallithea server to 0.6.3, and I was running 
into an issue where trying to view the landing page for one specific 
repository, but not others, was crashing.


I tracked this down to the changelog_table.html template. It runs 
message_lines = cs.message.splitlines() and tries to show the first 
line of the message by rendering message_lines[0]. However, if the 
commit message is completely empty, message_lines will be too, so 
trying to access message_lines[0] raises an uncaught IndexError.


Attached is a patch that does the bare minimum to avoid the crash, by 
forcing message_lines to contain an empty string in this case. It 
might be nice to do something nicer like render a dedicated, localized 
message in this case. But hopefully this helps identify the issue.


Hope this helps. If you need any additional information, please let me 
know.


Thanks,

--
Brett Smith

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: AW: Issue with kallithea 0.6.2 in combination with mercurial hook

[Mads Kiilerich]

The update hook is built-in functionality. I don't think it makes sense 
to make it customizable. If you want it to do something else, just 
disable "Update repository after push (hg update)" under VCS and add 
your own custom hook instead.


I agree that the built-in hook should use the virtualenv hg.

But also, it seems like the root cause of your problem is that your 
system has a bad hg in $PATH. It should work if your system hg works.


It seems very odd that &2 should break 
anything. At most, it can expose/hide other breakage.


/Mads


On 12/4/20 10:16 AM, Benny Bürger wrote:


I got the same error when running my own script *with* Kallithea (but 
this has already been fixed by using 

The current error occurs in the internal hook from Kallithea 
*changegroup.update*


But unfortunately I cannot change the hook because it is read-only.

I will give it a try with virtualenv, but in my opinion the hook 
should be changeable


*Von:*Mads Kiilerich 
*Gesendet:* Freitag, 4. Dezember 2020 00:43
*An:* Benny Bürger ; kallithea-general@sfconservancy.org
*Betreff:* Re: Issue with kallithea 0.6.2 in combination with 
mercurial hook


If you can get the same error when launching your scripts manually 
without Kallithea, then the problem must be in your local system - not 
directly related to Kallithea.


Again, I can only advise against installing in /usr/local - it is very 
easy to get inconsistent installations, and very hard to clean up.


But also, I guess Kallithea should do something to ensure that we 
actually invoke the hg installed in a Kallithea virtualenv. You can do 
the same in your scripts.


/Mads

On 12/1/20 11:24 AM, Benny Bürger wrote:

Sorry, just noticed it was the wrong hook.

Actually the changegroup.update fails (hg update >&2)

I guess the >&2 is not working.

In my own script I did a hg clone and I had to add a  <mailto:m...@kiilerich.com>
*Gesendet:* Montag, 30. November 2020 17:59
*An:* Benny Bürger  <mailto:buer...@simba.de>;
kallithea-general@sfconservancy.org
<mailto:kallithea-general@sfconservancy.org>
*Betreff:* Re: Issue with kallithea 0.6.2 in combination with
mercurial hook

On 11/30/20 5:30 PM, Benny Bürger wrote:

Hi,

I use the pretxnchangegroup.check with a bash script.

The script executes but the error code is ignored because of
the following error:

Gegenseite: Traceback (most recent call last):

Gegenseite:   File "/usr/local/bin/hg", line 43, in 

Gegenseite: dispatch.run()

Gegenseite:   File "/usr/lib/python3.8/importlib/util.py",
line 245, in __getattribute__

Gegenseite: self.__spec__.loader.exec_module(self)

Gegenseite:   File "",
line 783, in exec_module

Gegenseite:   File "", line 219,
in _call_with_frames_removed

Gegenseite:   File
"/usr/local/lib/python3.8/dist-packages/mercurial/dispatch.py",
line 22, in 

Gegenseite: from .i18n import _

Gegenseite:   File "/usr/lib/python3.8/importlib/util.py",
line 245, in __getattribute__

Gegenseite: self.__spec__.loader.exec_module(self)

Gegenseite:   File
"/usr/local/lib/python3.8/dist-packages/mercurial/i18n.py",
line 15, in 

Gegenseite: from .pycompat import getattr

Gegenseite: ImportError: cannot import name 'getattr' from
'mercurial.pycompat'
(/usr/local/lib/python3.8/dist-packages/mercurial/pycompat.py)

Environment: Linux repository 5.4.0-48-generic #52-Ubuntu SMP
Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I used pip to install Kallithea and all dependencies…

Is the hook working when launched manually with python3?

It looks like a problem in the installation of /usr/local/bin/hg .
/usr/local/lib/python3.8/dist-packages/mercurial/pycompat.py must
be a version less than 5.2 if it doesn't have getattr . That one
year old version didn't have good support for Python 3. I suggest
using latest Mercurial version. Python3 support is still improving.

How was Kallithea installed? Using virtualenv?

How was /usr/local/bin/hg installed?

Instead of installing in /usr/local , I would suggest using the hg
in the Kallithea virtualenv.

/Mads



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Issue with kallithea 0.6.2 in combination with mercurial hook

[Mads Kiilerich]

If you can get the same error when launching your scripts manually 
without Kallithea, then the problem must be in your local system - not 
directly related to Kallithea.


Again, I can only advise against installing in /usr/local - it is very 
easy to get inconsistent installations, and very hard to clean up.


But also, I guess Kallithea should do something to ensure that we 
actually invoke the hg installed in a Kallithea virtualenv. You can do 
the same in your scripts.


/Mads


On 12/1/20 11:24 AM, Benny Bürger wrote:


Sorry, just noticed it was the wrong hook.

Actually the changegroup.update fails (hg update >&2)

I guess the >&2 is not working.

In my own script I did a hg clone and I had to add a order to work, otherwise I would get the same error.


Unfortunately I cannot change the update command because it is read-only.

I already have mercurial 5.4 installed, although did a test with 5.6 
but there was another error.


*Von:*Mads Kiilerich 
*Gesendet:* Montag, 30. November 2020 17:59
*An:* Benny Bürger ; kallithea-general@sfconservancy.org
*Betreff:* Re: Issue with kallithea 0.6.2 in combination with 
mercurial hook


On 11/30/20 5:30 PM, Benny Bürger wrote:

Hi,

I use the pretxnchangegroup.check with a bash script.

The script executes but the error code is ignored because of the
following error:

Gegenseite: Traceback (most recent call last):

Gegenseite:   File "/usr/local/bin/hg", line 43, in 

Gegenseite: dispatch.run()

Gegenseite:   File "/usr/lib/python3.8/importlib/util.py", line
245, in __getattribute__

Gegenseite: self.__spec__.loader.exec_module(self)

Gegenseite:   File "", line
783, in exec_module

Gegenseite:   File "", line 219, in
_call_with_frames_removed

Gegenseite:   File
"/usr/local/lib/python3.8/dist-packages/mercurial/dispatch.py",
line 22, in 

Gegenseite: from .i18n import _

Gegenseite:   File "/usr/lib/python3.8/importlib/util.py", line
245, in __getattribute__

Gegenseite: self.__spec__.loader.exec_module(self)

Gegenseite:   File
"/usr/local/lib/python3.8/dist-packages/mercurial/i18n.py", line
15, in 

Gegenseite: from .pycompat import getattr

Gegenseite: ImportError: cannot import name 'getattr' from
'mercurial.pycompat'
(/usr/local/lib/python3.8/dist-packages/mercurial/pycompat.py)

Environment: Linux repository 5.4.0-48-generic #52-Ubuntu SMP Thu
Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I used pip to install Kallithea and all dependencies…

Is the hook working when launched manually with python3?

It looks like a problem in the installation of /usr/local/bin/hg . 
/usr/local/lib/python3.8/dist-packages/mercurial/pycompat.py must be a 
version less than 5.2 if it doesn't have getattr . That one year old 
version didn't have good support for Python 3. I suggest using latest 
Mercurial version. Python3 support is still improving.


How was Kallithea installed? Using virtualenv?

How was /usr/local/bin/hg installed?

Instead of installing in /usr/local , I would suggest using the hg in 
the Kallithea virtualenv.


/Mads



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: post-receive webhooks for git

[Mads Kiilerich]

On 12/2/20 4:49 AM, Ed Wong wrote:

Hi,

I'm using version 0.5.0 (due to system still using python.27).



I think all OSes make it easy to install Python 3 side by side with 
Python 2 without any further impact. I suggest upgrading. We don't have 
resources to support old versions.




I've been looking at the documentation for hooks and have looked at the
Admin->Settings->Hooks; but I can't make heads or tails in how
to set up a hook which gets triggered when someone pushes to the
repo.


I see that in the main repo there is a hooks dir with the post-receive
hook but with the "don't customize it manually" warning.

Where do I make changes to the post-receive hook?



The hooks support is a bit half baked.

Primarily, Kallithea use hooks internally to achieve some functionality. 
Modifying these will conceptually be like modifying Kallithea itself, 
and they are thus quite hardcoded.


Are you using Mercurial? Then it is quite easy to add additinoal custom 
hooks of any kind through the UI. Thinking about it, I guess we should 
clarify in the UI that the hooks settings is for Mercurial only ...


The Git hook functionality only allows for one hook of each kind - and 
has other hook entry points and a different hook API than Mercurial. It 
is not feasible to make them equal. But see Tim Ooms' related proposal 
in 
https://lists.sfconservancy.org/pipermail/kallithea-general/2020q4/thread.html#3241 
.


Extensions ( 
https://kallithea.readthedocs.io/en/default/usage/customization.html#behavioral-customization-kallithea-extensions 
) allow some more high level hooks, more consistent across Git and 
Mercurial, but they are global and do not have any UI.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Issue with kallithea 0.6.2 in combination with mercurial hook

[Mads Kiilerich]

On 11/30/20 5:30 PM, Benny Bürger wrote:


Hi,

I use the pretxnchangegroup.check with a bash script.

The script executes but the error code is ignored because of the 
following error:


Gegenseite: Traceback (most recent call last):

Gegenseite:   File "/usr/local/bin/hg", line 43, in 

Gegenseite: dispatch.run()

Gegenseite:   File "/usr/lib/python3.8/importlib/util.py", line 245, 
in __getattribute__


Gegenseite: self.__spec__.loader.exec_module(self)

Gegenseite:   File "", line 783, 
in exec_module


Gegenseite:   File "", line 219, in 
_call_with_frames_removed


Gegenseite:   File 
"/usr/local/lib/python3.8/dist-packages/mercurial/dispatch.py", line 
22, in 


Gegenseite: from .i18n import _

Gegenseite:   File "/usr/lib/python3.8/importlib/util.py", line 245, 
in __getattribute__


Gegenseite: self.__spec__.loader.exec_module(self)

Gegenseite:   File 
"/usr/local/lib/python3.8/dist-packages/mercurial/i18n.py", line 15, 
in 


Gegenseite: from .pycompat import getattr

Gegenseite: ImportError: cannot import name 'getattr' from 
'mercurial.pycompat' 
(/usr/local/lib/python3.8/dist-packages/mercurial/pycompat.py)


Environment: Linux repository 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 
10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


I used pip to install Kallithea and all dependencies…



Is the hook working when launched manually with python3?

It looks like a problem in the installation of /usr/local/bin/hg . 
/usr/local/lib/python3.8/dist-packages/mercurial/pycompat.py must be a 
version less than 5.2 if it doesn't have getattr . That one year old 
version didn't have good support for Python 3. I suggest using latest 
Mercurial version. Python3 support is still improving.


How was Kallithea installed? Using virtualenv?

How was /usr/local/bin/hg installed?

Instead of installing in /usr/local , I would suggest using the hg in 
the Kallithea virtualenv.


/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: problem with git push

[Mads Kiilerich]

On 11/20/20 9:49 AM, Łukasz Michalski wrote:

On 19/11/2020 20.53, Mads Kiilerich wrote:

On 11/19/20 4:31 PM, Łukasz Michalski wrote:

Hi,

I have a problem with one of git repos. When I try to push changes I got "remote 
rejected":

[zork@serenity filebench]$ git push zork
Password for 'https://zork@':
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 3.17 KiB | 1.58 MiB/s, done.
Total 8 (delta 7), reused 0 (delta 0), pack-reused 0
To https://
   ! [remote rejected] branch -> branch (pre-receive hook declined)
error: failed to push some refs to 'https://'

Kallithea  0.6.2 on Arch Linux.

What can I do to resolve this issue? I already tried to increase log level but 
I cannot find any clues in logs.


It works for other repos - just not this one?

The pre-receive hook is really not doing anything and should never reject 
pushes. I guess the hooks (for this repo?) isn't installed correctly? Inspect 
and compare hooks/post-receive for working and non-working repos. Or try to 
reinstall all hooks in Admin/Settings/Remap and Rescan/Install Git hooks + 
Overwrite existing Git hooks.

It might be a very fundamental problem (such as finding the right Python 
interpreter to run the hook) so I doubt Kallithea logging could say anything. 
But perhaps the web server error log has something.

/Mads


Rescan+Overwrite helped. Thanks!

I also upgraded kallithea to 0.6.2 changing python version from 3.7 to 3.8 (by 
creating new venv).



Great it works.

Should the documentation be improved to clarify the importance of 
re-installing git hooks after upgrading Kallithea or Python? Where and 
how would you suggest phrasing it?


I don't know how we can improve this handling. We rely on Git 
functionality. If you modify hooks/pre-receive to invoke sys.stdin.write 
or sys.stderr.write or cause an exception (for example by adding an 
invalid import), you will see that as "remote:" lines on the client 
side. But if you modify the first line with #! to point at something 
that doesn't work, Git will be silent - not even a "bad interpreter" 
message as the shell would do.


We could perhaps change the hooks to *always* write something, such as 
"rejected by Kallithea hook" or "accepted by Kallithea hook" ... and if 
there is no such message, we can know that the hook failed badly. But 
that wouldn't be explicit and obvious ... and we did figure it out anyway.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: problem with git push

[Mads Kiilerich]

On 11/19/20 4:31 PM, Łukasz Michalski wrote:

Hi,

I have a problem with one of git repos. When I try to push changes I got "remote 
rejected":

[zork@serenity filebench]$ git push zork
Password for 'https://zork@':
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 8 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 3.17 KiB | 1.58 MiB/s, done.
Total 8 (delta 7), reused 0 (delta 0), pack-reused 0
To https://
  ! [remote rejected] branch -> branch (pre-receive hook declined)
error: failed to push some refs to 'https://'

Kallithea  0.6.2 on Arch Linux.

What can I do to resolve this issue? I already tried to increase log level but 
I cannot find any clues in logs.



It works for other repos - just not this one?

The pre-receive hook is really not doing anything and should never 
reject pushes. I guess the hooks (for this repo?) isn't installed 
correctly? Inspect and compare hooks/post-receive for working and 
non-working repos. Or try to reinstall all hooks in Admin/Settings/Remap 
and Rescan/Install Git hooks + Overwrite existing Git hooks.


It might be a very fundamental problem (such as finding the right Python 
interpreter to run the hook) so I doubt Kallithea logging could say 
anything. But perhaps the web server error log has something.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Bug: ssl error with postgres and celery

[Mads Kiilerich]

On 11/5/20 1:11 AM, Mads Kiilerich wrote:

On 11/3/20 5:08 PM, Valentin Kleibel wrote:

Hi,

We are using kallithea with a postgres database on another server 
connected via ssl.
It seems that celery in version 4 uses prefork in its worker pool 
model now, which leads to issues with such a database connection: 


I think you are right something is wrong.

First, the celery-run will not only take a config file as parameter 
and read it. It will also initialize the WSGI app and database 
connection, before calling into celery. Effectively making it 
pre-fork. I think you are right that we only should initialize things 
after celery has forked and started worker processes.



The first changes on 
https://kallithea-scm.org/repos/kallithea/pull-request/292/_/celery_and_stuff 
do this. After some refactorings (that may or may not be essential for 
the change), I think the proper fix is in 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/e32c37b84c53bc6342e7b4698cb06168a7bcff15 
.


I don't think we should have to dispose the engine. We just shouldn't 
initialize it pre-fork.


Does this work for you?

Disposing the engine for each task seems inefficient and shouldn't be 
necessary. As an alternative to your patch for disposing in dbsession, 
it could perhaps just dispose in 
kallithea.CELERY_APP.loader.on_worker_process_init .



But also, it might be a bit odd that we have the celery-run entry 
point. It would perhaps be better to somehow use "celery" as entry 
point and let it call into Kallithea in each worker. But I haven't 
investigated how feasible that would be.



This is still TODO.

There is still a lot of room for improvement, making sure that we use 
Celery in a simple and recommended way, avoiding the need for 
initializing Celery at *import* time, and upgrading to Celery 5. 
Contributions in this area would be welcome ;-)


/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Bug: ssl error with postgres and celery

[Mads Kiilerich]

On 11/3/20 5:08 PM, Valentin Kleibel wrote:

Hi,

We are using kallithea with a postgres database on another server 
connected via ssl.
It seems that celery in version 4 uses prefork in its worker pool 
model now, which leads to issues with such a database connection: 


I think you are right something is wrong.

First, the celery-run will not only take a config file as parameter and 
read it. It will also initialize the WSGI app and database connection, 
before calling into celery. Effectively making it pre-fork. I think you 
are right that we only should initialize things after celery has forked 
and started worker processes.


But also, it might be a bit odd that we have the celery-run entry point. 
It would perhaps be better to somehow use "celery" as entry point and 
let it call into Kallithea in each worker. But I haven't investigated 
how feasible that would be.


As a workaround, can you use celery.worker_concurrency = 1 but launch 
celery-run several times?


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: Bug: invalid headers for git over http

[Mads Kiilerich]

Hi

Thanks for reporting ... and for the proposed fix.

That bug is something I missed when porting to py3. Apparently, other 
Git implementations are less strict, and it didn't come up while testing.


But I think the fix can be made simpler and more complete:
https://kallithea-scm.org/repos/kallithea-incoming/changeset/b9be708625087a6618240d614d27782632a80b2b

Can you confirm this works as well for you?

/Mads


On 10/28/20 2:13 PM, Valentin wrote:

Hi,

I've noticed cases in which kallithea returns headers that seem 
incorrect on git requests over http.
This first occurred as some users had problems fetching repos from our 
kallithea instance with software that uses the JGit library on Windows 
(user agent string: JGit/5.4.2.201908231537-r).

The error message noted:
'git fetch' command failed (repository dir: dir>/system/caches/git/git-233054F9.git).
stderr: https:: expected Content-Type 
application/x-git-upload-pack-result; received Content-Type 
application/x-b'git-upload-pack'-result


I was able to reproduce this header by using git for windows, although 
it didn't produce an error:

valentin@CASHEW MINGW64 ~
$ GIT_CURL_VERBOSE=1 git clone https:
Cloning into 'master'...

...
...
12:14:51.522612 http.c:721  <= Recv header, 59 
bytes (0x003b)
12:14:51.522612 http.c:733  <= Recv header: Content-Type: 
application/x-git-upload-pack-advertisement

...
...
12:15:32.147657 http.c:721  <= Recv header, 55 
bytes (0x0037)
12:15:32.147657 http.c:733  <= Recv header: Content-Type: 
application/x-b'git-upload-pack'-result



The problem arises in pygrack.py [0] where encode('utf-8') is used on 
line 178:
resp.content_type = 'application/x-%s-result' % 
git_command.encode('utf-8')


in contrary, on line 128 str() is used:
resp.content_type = 'application/x-%s-advertisement' % str(git_command)

I patched my kallithea instance to use str() (see the attached diff) 
which resolves the problem.
I think this is safe to do, as git_command is checked against valid 
commands before and therefor must be convertible to a string.


Hope i got that right and you can integrate this into an upcoming 
release.


Cheers,
Valentin


[0] 
https://kallithea-scm.org/repos/kallithea/files/855b37d3bacdc6175566ca7d23c19e2352da1087/kallithea/lib/middleware/pygrack.py



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: outdated links in mailman welcome message

[Mads Kiilerich]

Hi

Thanks for reporting.

I have fixed the welcome message.

Enjoy,
Mads


On 10/27/20 3:47 PM, Valentin wrote:

Hi,

I just joined thje mailing list and noticed that the nmailman welcome 
message for this list still contains links to the removed wiki and 
issue tracker on bitbucket.org.
It would be great if you could replace those links with updated 
information.


Thank you,
Valentin
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH] git: move non-kallithea hooks and execute other hooks

[Mads Kiilerich]

On 10/23/20 9:30 AM, Tim Ooms wrote:

# HG changeset patch
# User Tim Ooms 
# Date 1603201903 -7200
#  Tue Oct 20 15:51:43 2020 +0200
# Node ID 672e57b165d0c1774b692b5706a174bf98f42e4c
# Parent  b9b53e25a08d3714c54d82641b419e6d01820e12
git: move non-kallithea hooks and execute other hooks


It might have rough edges to bypass Kallithea when accessing 
Kallithea-managed repos. It is perhaps better to go with the 
assumption that Kallithea fully owns the repositories it manage.


I tried to not bypass Kallithea by letting it execute the other hooks. 
I know the mercurial hooks are more integrated, but as the git hooks 
don't work the same way I think this is the next best thing. Although 
there may be better ways, I'm not sure.



If you only access the repos through Kallithea, I think it would be 
better to run your hooks as Kallithea extensions. I'm not sure that's 
possible right now, but we can perhaps fix that.



Can you say more about your use case? What other hooks are installed? 
What precautions do they use to avoid clobbering the Kallithea hooks?


Use cases I think of:
post:
* trigger CI build (like jenkins)
* notify bug tracker (like the bug genie)
pre:
* check for commit message
* verify syntax/codestyle
Other examples: 
https://scriptrunner.adaptavist.com/stash/3.0.14-beta1/docs/pre_receive_hooks/
These hooks are really independent and don't touch the repository. If 
they fail, it has no effect on the other hooks. A hook can mess up if 
it just hangs and does not return. But you will always have this if 
people are using custom hooks, even in mercurial.



Thanks. Some good examples of "hooks" into the default process.

I understand you install these hooks yourself - it is not some 3rd party 
system doing it. Your purpose with installing the hooks is to customize 
the Kallithea process.


In that case, would it perhaps be better to hook into Kallithea 
"extensions" instead of hooking in as Git hooks? If we don't have the 
right kind of extension hooks, perhaps we should introduce that?



It must be a general challenge with Git to fit multiple hooks into the 
single hook slot. Is there any prior art for solving / working around 
that problem?


Yes, there is:
* solution using "pee": 
https://git.seveas.net/using-multiple-post-receive-hooks.html



Nice. But depends on perl. It would be nice to have it as Python. But I 
guess Git already depends on perl ...



* using bash built-ins: 
https://serverfault.com/questions/909153/how-to-add-multiple-post-receive-hooks-to-a-git-repository



It seems like the hooks can have multiple lines on stdin. This batch 
script will invoke the hooks multiple times in that case. That might be 
unfortunate. And it also doesn't detect failures.


And unfortunately, it seems like they both just describe how it *can* be 
done. It is not a standard way of doing it.



It would be nice if there was a "standard" dispatcher (perhaps a 
simple shell script) that could be installed in The Hook Location - 
something that picked up multiple hook files. Instead of having our 
hook call others, it would perhaps be better to install a generic-ish 
hook and let it dispatch to both our hook and the existing one.


I would indeed like a shell solution (I used the bash solution till 
now, but till now that causes problems when doing an update), but that 
ain't portable and I think Kallithea can run on Windows. It can have 
advantages to let Kallithea run the other hooks: they can be managed 
(in a later stage) by Kallithea, can be logged by Kallithea and admins 
can be notified if a hook fails to run.
Also, running directly the Kallithea hook, Kallithea -being the one 
managing the repository- is sure that its hook is correctly called and 
it's shebang is correctly set and executable.


This can also be done using something generic-ish (and I like the 
idea), but you add another layer that can fail and another file to 
keep up to date like the existing hook.



As a step in the right direction, we can perhaps let Kallithea 
create/overwrite post-receive.d/kallithea.hook *if* post-receive already 
exists *and* isn't a kallithea hook, *and* if post-receive.d exists.


That will leave it to you to install the dispatcher when installing your 
hooks.



The solution proposed here also touch upon lessons learned from 
general /etc/xxx.rc/ configuration systems: People will often want to 
rename configuration files to something like '.bu' or '.old' or 
'.disabled'. It is thus a very good idea to only match on one file 
extension - and perhaps also a prefix. The "real" hooks do not have 
any extension, but the globbed ones could perhaps match 
'pre-receive.*.hook'? Also, we should check that they are executable 
for the current user before trying to run them.


Nice catches! That would indeed improve the robustness. A 
try-catch-log would probably also be in place, so we always try all 
hooks.
For pre-receive hooks, we should also check the return code of all 
hooks and return non-ze

Re: [PATCH] git: move non-kallithea hooks and execute other hooks

[Mads Kiilerich]

On 10/20/20 5:19 PM, Tim Ooms wrote:

# HG changeset patch
# User Tim Ooms 
# Date 1603201903 -7200
#  Tue Oct 20 15:51:43 2020 +0200
# Node ID 672e57b165d0c1774b692b5706a174bf98f42e4c
# Parent  b9b53e25a08d3714c54d82641b419e6d01820e12
git: move non-kallithea hooks and execute other hooks



Thanks for working on this.

It might have rough edges to bypass Kallithea when accessing 
Kallithea-managed repos. It is perhaps better to go with the assumption 
that Kallithea fully owns the repositories it manage.


Can you say more about your use case? What other hooks are installed? 
What precautions do they use to avoid clobbering the Kallithea hooks?


It must be a general challenge with Git to fit multiple hooks into the 
single hook slot. Is there any prior art for solving / working around 
that problem?


It would be nice if there was a "standard" dispatcher (perhaps a simple 
shell script) that could be installed in The Hook Location - something 
that picked up multiple hook files. Instead of having our hook call 
others, it would perhaps be better to install a generic-ish hook and let 
it dispatch to both our hook and the existing one.


The solution proposed here also touch upon lessons learned from general 
/etc/xxx.rc/ configuration systems: People will often want to rename 
configuration files to something like '.bu' or '.old' or '.disabled'. It 
is thus a very good idea to only match on one file extension - and 
perhaps also a prefix. The "real" hooks do not have any extension, but 
the globbed ones could perhaps match 'pre-receive.*.hook'? Also, we 
should check that they are executable for the current user before trying 
to run them.


For the chmod changes, I think we only should care about the hooks we 
install. If the existing hook isn't executable, git will ignore it and 
we should do the same - not try to clean things up we don't know about. 
We just wrote our own hook file, so we know it isn't a symlink, and 0755 
should be fine? Ideally, we should probably use umask and set X bit ... 
but I doubt that ever will make any difference.


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: adding ssh public keys with lf in it

[Mads Kiilerich]

On 9/28/20 2:03 AM, Ed Wong wrote:

Thanks for the report and patch.

Can you clarify exactly which problem you saw? Did you get a "nice"
"Incorrect SSH key - failed to decode base64 part" message when adding
the key? A correct and correctly shown message ... but not helpful? And
you missed the error message and thought the key had been added
successfully?

The key was added successfully without any errors.  When I reload the
account's ssh keys, it shows the added key.



Ok. I coldn't verify ... but that depends on whether the truncation was 
in a place that didn't make the base64 string invalid.


I implemented more thorough validation of ssh keys and ended up with 
https://kallithea-scm.org/repos/kallithea/pull-request/274/_/ssh_key_verification 
- how does that work for you?


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: adding ssh public keys with lf in it

[Mads Kiilerich]

(Note: please post to kallithea-general@sfconservancy.org - not to the 
-bounces address.)



Thanks for the report and patch.

Can you clarify exactly which problem you saw? Did you get a "nice" 
"Incorrect SSH key - failed to decode base64 part" message when adding 
the key? A correct and correctly shown message ... but not helpful? And 
you missed the error message and thought the key had been added 
successfully?



Some comments to the proposed fix:

Note that public keys can be added both by admin 
kallithea/controllers/admin/users.py and by users 
kallithea/controllers/admin/my_account.py . A fix of this kind would 
thus probably be better to have in shared code, for example in 
kallithea/model/ssh_key.py .


It seems like the added test will pass, also without the fix. The 
kallithea/model/db.py public_key setter will just decode using 
base64.b64decode which happens to happily ignore any kind of whitespace. 
I guess it would be better to test this change like we test ssh key 
adding in kallithea/tests/functional/test_admin_users.py and 
kallithea/tests/functional/test_my_account.py .



But back to the core of the problem:

The format of these ssh public keys is that they are one line. First the 
key type, then space, then the base64 encoded key, then optional space 
followed by anything that is a comment and ignored. I thus have some 
concerns of adding partial support for a non-standard format. Especially 
as this is security sensitive and we thus try to be paranoid. When 
parsing the base64 encoded part, we even have a "Incorrect SSH key - 
unexpected characters in base64 part" check to explicitly avoid newlines.


But we could perhaps do it anyway... Would 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/d8ec9261cead78bb6f4768ebf7f82bd21c0d74fd 
work for you and fix your problem?


/Mads



On 9/25/20 9:50 AM, Ed Wong wrote:

Hi, .

I created a new user in a local Kallithea instance and added a public
key to it and saved.

Then I proceeded to try to clone from a different system using that
user account's key.  No matter what I did, it kept on asking for
a password.
just
Completely stumped me as I could clone off a different account,
so I went to insspect the public key and realized that when
I copy and pasted off a mingw terminal, it had included
a bunch of \n.  I removed the \n and repasted the public_key.

Then it allowed me to clone.

I'm currently setting up a dev env to test if my fix is gonna
work though I suspect it's a bit of a hack.  I've included
it in this post for comment.


Edmund

diff -r c819a1e9103b kallithea/controllers/admin/users.py
--- a/kallithea/controllers/admin/users.py  Mon Aug 24 15:02:16 2020
+0200
+++ b/kallithea/controllers/admin/users.py  Fri Sep 25 15:33:19 2020
+0800
@@ -446,10 +446,13 @@

  @IfSshEnabled
  def ssh_keys_add(self, id):
+def rem_newline(in_pub_key):
+return in_pub_key.replace("\r\n", "").replace("\n", "")
+
  c.user = self._get_user_or_raise_if_default(id)

  description = request.POST.get('description')
-public_key = request.POST.get('public_key')
+public_key = rem_newline(request.POST.get('public_key'))
  try:
  new_ssh_key = SshKeyModel().create(c.user.user_id,
 description, public_key)
diff -r c819a1e9103b kallithea/tests/models/test_user_ssh_keys.py
--- a/kallithea/tests/models/test_user_ssh_keys.py  Mon Aug 24
15:02:16 2020 +0200
+++ b/kallithea/tests/models/test_user_ssh_keys.py  Fri Sep 25
15:33:19 2020 +0800
@@ -7,6 +7,10 @@

  public_key = 'ssh-rsa
B3NzaC1yc2EDAQABgQC6Ycnc2oUZHQnQwuqgZqTTdMDZD7ataf3JM7oG2Fw8JR6cdmz4QZLe5mfDwaFwG2pWHLRpVqzfrD/Pn3rIO++bgCJH5ydczrl1WScfryV1hYMJ/4EzLGM657J1/q5EI+b9SntKjf4ax+KP322L0TNQGbZUHLbfG2MwHMrYBQpHUQ==
kallithea@localhost'
+public_key_with_lf = 'ssh-rsa B3NzaC1yc2EDAQABAA\n' + \
+   'AAgQC6Ycnc2oUZHQnQwuqgZqTTdMDZD7ataf3\n' +
+
'JM7oG2Fw8JR6cdmz4QZLe5mfDwaFwG2pWHLRpVqzfrD/Pn3rIO++bgCJH5ydczrl1WScfryV1hYMJ/4EzLGM657J1/q5EI+b9SntKjf4ax+KP322L0TNQGbZUHLbfG2MwHMrYBQpHUQ==
kallithea@localhost'
+

  class TestUserSshKeys(TestController):

@@ -15,3 +19,9 @@
  key_model.public_key = public_key
  expected = 'Ke3oUCNJM87P0jJTb3D+e3shjceP2CqMpQKVd75E9I8'
  assert expected == key_model.fingerprint
+
+def test_line_feed_public_key(self):
+key_model = UserSshKeys()
+key_model.public_key = public_key_with_lf
+expected = 'Ke3oUCNJM87P0jJTb3D+e3shjceP2CqMpQKVd75E9I8'
+assert expected == key_model.fingerprint



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [PATCH fixes dulwich exception]

[Mads Kiilerich]

I started looking at it. 
https://kallithea-scm.org/repos/kallithea/pull-request/273/_/refactorings_and_stable_fixes_from_mailing_list 
propose some tests (after refactoring to add basic test coverage of git 
PRs).


/Mads


On 9/25/20 6:38 PM, Thomas De Schampheleire wrote:

Hi,

Thanks for your patch.
I did not yet have time to verify and test it.

Do you think it would be possible for you to add a test for the 
failing behaviour?


Thanks,
Thomas

On Mon, Sep 21, 2020, 15:22 tonich sh > wrote:


second try:
Fixes dulwich exception on pull-request deletion
diff -r a2ec23356e6b kallithea/model/pull_request.py
--- a/kallithea/model/pull_request.py   Tue Sep 15 21:30:26 2020 +0200
+++ b/kallithea/model/pull_request.py   Mon Sep 21 16:14:46 2020 +0300
@@ -139,7 +139,7 @@
         if pull_request.org_repo.scm_instance.alias == 'git':
             # remove a ref under refs/pull/ so that commits can be
garbage-collected
             try:
-                del
pull_request.org_repo.scm_instance._repo["refs/pull/%d/head" %
pull_request.pull_request_id]
+                del
pull_request.org_repo.scm_instance._repo[b"refs/pull/%d/head" %
pull_request.pull_request_id]
             except KeyError:
                 pass

пн, 21 сент. 2020 г. в 16:17, tonich sh mailto:tonich...@gmail.com>>:

>
> diff -r a2ec23356e6b kallithea/model/pull_request.py
> --- a/kallithea/model/pull_request.py   Tue Sep 15 21:30:26 2020
+0200
> +++ b/kallithea/model/pull_request.py   Mon Sep 21 16:14:46 2020
+0300
> @@ -139,7 +139,7 @@
>          if pull_request.org_repo.scm_instance.alias == 'git':
>              # remove a ref under refs/pull/ so that commits can be
> garbage-collected
>              try:
> -                del
> pull_request.org_repo.scm_instance._repo["refs/pull/%d/head" %
> pull_request.pull_request_id]
> +                del
> pull_request.org_repo.scm_instance._repo[b"refs/pull/%d/head" %
> pull_request.pull_request_id]
>              except KeyError:
>                  pass



--
С уважением, Антон Щур.
___
kallithea-general mailing list
kallithea-general@sfconservancy.org

https://lists.sfconservancy.org/mailman/listinfo/kallithea-general


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general



___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: BUG: Cannot browse files

[Mads Kiilerich]

On 8/23/20 7:29 AM, Thomas De Schampheleire wrote:


Aug 22 22:29:49 universe gearbox[22393]:   File

"/srv/kallithea/venv/lib/python3.7/site-packages/kallithea/lib/vcs/nodes.py",

line 587, in __init__
Aug 22 22:29:49 universe gearbox[22393]:     self.path =
name.rstrip('/')
Aug 22 22:29:49 universe gearbox[22393]: TypeError: a bytes-like
object
is required, not 'str'



I think this is the problem fixed by following commit on the stable 
branch:


https://kallithea-scm.org/repos/kallithea/changeset/f48b12755d83cba1c8e6613e9df2b004ade3721f


There is no release yet with that included, but you could either 
install from the repo, or patch locally.



We never got a reproducible test case for the report that caused the 
fix. It would be nice if you could confirm it solves your problem.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [success]

[Mads Kiilerich]

On 8/15/20 2:00 PM, Uwe Brauer wrote:


 1. I understand kallithea comes with some sort of notifying utility?
That would be sort of important, I usually use notify myself but
other users might not.



Yes, users will be notified by mail when they are added to PRs, and also 
when comments are added to PRs they are involved with or on changesets 
they created.




 2. Can the user configure his/hers repository to be non publishing?
In bitbucket you could, in helix you can't for example.



It can currently only be done by the sysadmin, by adding "[phases]" 
"publish = False" to .hg/hgrc of the repo.


Needless to say, the functionality (and UI) for letting the user control 
it could be added ...




 3. Would it be useful to write something up: how to install
kallithea for Ubuntu 16.04, for example, if you want I could give
it a try if you tell me in which format, markdown restructured
text etc etc



I don't know.

It seems like most of your problems were quite unique. In a not-so-sweet 
spot between not following the documentation (or not reading our unclear 
documentation as we intended), and also needing more detailed 
documentation due to less python or sysadmin background and insufficient 
research of new areas and problems. Our documentation must be generic in 
order to stay focused and usable for all users, so I don't know if it is 
feasible to try to cover everything that would have helped.


Summarizing some of the problems (and hinting to root causes even though 
it might seem like "blaming"): Using a 4 years old LTS (with two more 
recent LTS releases), trying to use a python3 version that explicitly 
isn't supported, then facing problems with python 3 migration and node 
packaging in that particular distro and version - already fixed in later 
distro versions. Using a less common shell without knowing how that 
shell is "special". Trying to install as root, which generally is risky 
and something to avoid. Not following the advises but making other 
assumptions - like trying to uninstall the system python, not using 
virtualenv, then not using virtualenv as recommended, and assumptions of 
similarity with lisp. And then the dead end of trying python 2.


But it would be interesting to see your take on the root causes and how 
they can be mitigated. Perhaps try to repeat the installation on another 
system.


Thanks for helping making the software better.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: installation still does work

[Mads Kiilerich]

On 8/14/20 10:03 PM, Uwe Brauer wrote:

> Hi Uwe,
> On Fri, Aug 14, 2020, 17:24 Uwe Brauer  wrote:


> In general you should avoid changing such links manually. I suggest you
> restore the original one.
> The virtual environment (venv) created with the command will use the 
python
> used for the command itself. This means that you will get a python 3.8 env
> if you create it with python 3.8.

> So use the following command:

> python3.8 -m venv ../kallithea-venv

Thanks this does not work, because python3.8-venv is missing



How did you conclude this package was missing? venv is a part of the 
Python 3 standard library and should be available in all Python 3 
installations.


According to 
https://packages.ubuntu.com/eoan/amd64/python3.8-venv/filelist , a 
package by that name only contains a standalone command that we don't 
recommend.



/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: [Python 3.6, 3.7 3.8]

[Mads Kiilerich]

On 8/13/20 9:24 PM, Uwe Brauer wrote:

"MK" == Mads Kiilerich  writes:

> On 8/13/20 8:53 PM, Uwe Brauer wrote:
>>> I suggest creating /srv/kallithea as root, and chown it to the
>>> kallithea user. (I am trying to tweak the documentation based on your
>>> input and already had some draft changes to this.)
>> Good, at least something useful will come out from my intents. ;-)
>>
>>
>> When you say kallithea user, do you mean me, or do you mean that I shall
>> create a user (and group) named kallithea?


It seems possible to install python 3.6/3.7/3.8 on Ubuntu 16.04.

I'd say newer is better, but maybe this is not true, any
recommendations?



They should all work fine. 3.8 is quite mature now. 3.6 might get less 
testing these days and might start bitrotting.


These days I usually use a 3.9 rc for development and testing.

/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: how to install kallithea from source

[Mads Kiilerich]

On 8/13/20 9:15 PM, Uwe Brauer wrote:
which version (changeset) requires only 3.5)?? 



3.6 was the first python3 we supported. (Inherited from the versions 
supported by Mercurial).


3.5 was never supported.

0.5.2 aa0a637fa6f6 was the last py2 version.


For a simple test I think that should be sufficient, buy only if you
don't tell me that such a kallithea version lacks a lot of important
features.



I don't think there have been any big new features since. Mainly a lot 
of maturity changes.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: how to install kallithea from source

[Mads Kiilerich]

On 8/13/20 8:53 PM, Uwe Brauer wrote:

I suggest creating /srv/kallithea as root, and chown it to the
kallithea user. (I am trying to tweak the documentation based on your
input and already had some draft changes to this.)

Good, at least something useful will come out from my intents. ;-)


When you say kallithea user, do you mean me, or do you mean that I shall
create a user (and group) named kallithea?



Yes, I recommend running Kallithea as a separate user to control how it 
can interact with other system users.


You *could* run Kallithea as you (especially if evaluating or 
developing), but for production it should be a separate user.




Other than creating the installation directory (which could be
elsewhere), there should not be any global or .local changes. Just
active the virtualenv first (or run binaries in the virtualenv's bin
directory).


Well I tried to do this, but it failed again.

Here is what I did, using bash

  sudo mkdir /srv/kallithea
  sudo chown -R oub /srv/kallithea
  sudo chgrp -R oub /srv/kallithea
  cd $HOME/src/kallithea
python3 -m venv /srv/kallithea/venv
. /srv/kallithea/venv/bin/activate
pip install --upgrade pip setuptools
pip install --upgrade kallithea

,
| ERROR: Command errored out with exit status 1:
|
|  command: /srv/kallithea/venv/bin/python3 -c 'import sys,
|  setuptools, tokenize; sys.argv[0] =
|  '"'"'/tmp/pip-install-ydr3jegw/kallithea/setup.py'"'"';
|  
__file__='"'"'/tmp/pip-install-ydr3jegw/kallithea/setup.py'"'"';f=getattr(tokenize,
|  '"'"'open'"'"',
|  open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
|  '"'"'\n'"'"');f.close();exec(compile(code, __file__,
|  '"'"'exec'"'"'))' egg_info --egg-base
|  /tmp/pip-pip-egg-info-3uwu0aad
|
|  cwd: /tmp/pip-install-ydr3jegw/kallithea/
| Complete output (5 lines):
| Traceback (most recent call last):
|   File "", line 1, in 
|   File "/tmp/pip-install-ydr3jegw/kallithea/setup.py", line 13, in 

| raise Exception('Kallithea requires Python 3.6 or later')
| Exception: Kallithea requires Python 3.6 or later
| 
| ERROR: Command errored out with exit status 1: python setup.py egg_info Check 
the logs for full command output.
`



DOH. Now i notice what I think Thomas mentioned earlier. Kallithea does 
indeed require Python 3.6 or later. And it seems like your system is 
using 3.5?


Which Ubuntu version are you using now? I guess they have later Python 
versions available as standard packages, properly namespaced to avoid 
conflicts. If creating the virtualenv with for example python3.8, your 
/srv/kallithea/venv/bin/python3 and everything in the venv will use that 
too.


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: how to install kallithea from source

[Mads Kiilerich]

On 8/13/20 7:54 PM, Uwe Brauer wrote:

> Hi Uwe,
> On Thu, Aug 13, 2020, 17:50 Uwe Brauer 
 wrote:


> The '.' is an alias to the 'source' shell command. It reads and executes
> the file provided as arguments.
> See section "Shell Builtin Commands" at:
> https://linux.die.net/man/1/bash

Ah this is a bash thing, I have used only tcsh all my life and haven't
come across that.



Ok. If using less common choices, some documentation will not apply 
directly ;-)


https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments 
hints that you might be able to source activate.csh instead. Or just use 
bash for this.




But I still don't understand


cd kallithea  #OK
python3 -m venv ../kallithea-venv #OK
. ../kallithea-venv/bin/activate #OK
pip install --upgrade pip setuptools # this is system wide



No, it is not system wide. When the virtualenv is activated, it will 
modify PATH to start with ../kallithea-venv/bin/ where it's own pip (and 
python) "binaries" are available - they will work inside the virtualenv, 
not globally.


Since you are not running as root, there is also no way it could have 
modified anything system wide.




pip install --upgrade -e . # this is system wide
python3 setup.py compile_catalog   # for translation of the UI

Now I get
pip install --upgrade -e . # this is system wide
ERROR: File "setup.py" not found. Directory cannot be installed in editable 
mode: /home/oub/src



Are you still in the directory of the kallithea source (with a setup.py)?



pip install --upgrade -e .
,
| Obtaining file:///home/oub/src/kallithea
| Complete output from command python setup.py egg_info:
| Traceback (most recent call last):
|   File "", line 1, in 
|   File "/home/oub/src/kallithea/setup.py", line 13, in 
| raise Exception('Kallithea requires Python 3.6 or later')
| Exception: Kallithea requires Python 3.6 or later
`

Ok so this is impossible. Sigh



No, it is not impossible. You just didn't activate the virtualenv - thus 
the later commands doesn't work as intended. Either use bash, or 
activate your csh.




Then I checked the documentation about virtualenv

python3 -m venv /srv/kallithea/venv

Gives me an error
Permission denied: '/srv/kallithea'

So shall I run this as root?



I suggest creating /srv/kallithea as root, and chown it to the kallithea 
user. (I am trying to tweak the documentation based on your input and 
already had some draft changes to this.)




I am a bit confused about who (root or normal user) to install and
where, globally in the system or in .local



Other than creating the installation directory (which could be 
elsewhere), there should not be any global or .local changes. Just 
active the virtualenv first (or run binaries in the virtualenv's bin 
directory).


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: how to install kallithea from source

[Mads Kiilerich]

On 8/13/20 5:45 PM, Uwe Brauer wrote:

Hi

Since I cannot install kallitea via pip (my python version in Ubuntu
16.04 is too old) I thought to give the installation from source a try.


The documentation in
https://kallithea.readthedocs.io/en/stable/installation.html

Reads

,
| hg clone https://kallithea-scm.org/repos/kallithea -u stable
| cd kallithea
| python3 -m venv ../kallithea-venv
| . ../kallithea-venv/bin/activate
| pip install --upgrade pip setuptools
| pip install --upgrade -e .
| python3 setup.py compile_catalog   # for translation of the UI
`

I don't understand the lines

  python3 -m venv ../kallithea-venv
  . ../kallithea-venv/bin/activate

Is this supposed to be one line?
What does the . At the beginning of the second line supposed to be?


No, that is two separate commands that can be run as written.

`.` is a `sh` command for sourcing another file. In recent derivates 
like bash it can also be invoked as `source`. See `help .` and `help 
source` and 
https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments 
(and https://kallithea-scm.org/repos/kallithea/changeset/855ba1f07aeb ).


/Mads



/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: cannot set up Kallithea in Ubuntu 16.04

[Mads Kiilerich]

On 8/13/20 5:42 PM, Uwe Brauer wrote:

> It seems like your installation somehow got off track and you thus got
> on a bumpy road. At this point I would suggest reinstalling Ubuntu (to
> get an easy cleanup after the things done as root) and start over,
> keeping records of which choices and commands you use - that will make
> it easier to help if you should encounter problems again.

Right now I cannot reinstall (or better said upgrade to 20.04) my Ubuntu
since I cannot backup my home directory



That sounds like other and severe problems. It would perhaps be better 
to fix that first.




My last attempt to sort out this mess.

sudo -H pip uninstall kallisthea

And off it is from /usr/local/lib/python2.7
pip install --user kallithea

Gives me now

Using cached Kallithea-0.6.1.tar.gz (4.3 MB)
,
|
| ERROR: Command errored out with exit status 1:
|  command: /usr/bin/python -c 'import sys, setuptools, tokenize;
|  sys.argv[0] = '"'"'/tmp/pip-install-MIEe7A/kallithea/setup.py'"'"';
|  
__file__='"'"'/tmp/pip-install-MIEe7A/kallithea/setup.py'"'"';f=getattr(tokenize,
|  '"'"'open'"'"',
|  open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
|  '"'"'\n'"'"');f.close();exec(compile(code, __file__,
|  '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-zO3AGs
|  cwd: /tmp/pip-install-MIEe7A/kallithea/
| Complete output (5 lines):
| Traceback (most recent call last):
|   File "", line 1, in 
|   File "/tmp/pip-install-MIEe7A/kallithea/setup.py", line 13, in 
| raise Exception('Kallithea requires Python 3.6 or later')
| Exception: Kallithea requires Python 3.6 or later
| 
| ERROR: Command errored out with exit status 1: python setup.py egg_info Check 
the logs for full command output.
`


And Ubuntu 16.04 comes with python 3.5 so end of game, maybe I could try
to install it from source



We don't know how you installed Kallithea before, but you must have been 
using a py3 pip. Weird it doesn't pick that one up now. Something weird 
is going on.


It seems like you ended up using the last "Installing a released version 
without virtualenv" options on 
https://kallithea.readthedocs.io/en/stable/installation.html . I had 
forgotten we mention these in the documentation. That doesn't get much 
testing. A Kallithea specific virtualenv has the advantage of keeping 
everything installed for Kallithea in one place, separate from things 
that might be installed for other reasons.


/Mads
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general