migrate repository from mercurial to git
Hi, I would like to migrate a repository myrepo from mercurial to git. I want to rename mercurial repository to myrepo_old and use orginal name for new git repository. Is it enough to rename mercurial repository on disk and copy new one under original name? Old one can have some closed pull requests. Thanks, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: problem with git push
On 20/11/2020 15.42, Mads Kiilerich wrote: > On 11/20/20 9:49 AM, Łukasz Michalski wrote: >> Rescan+Overwrite helped. Thanks! >> >> I also upgraded kallithea to 0.6.2 changing python version from 3.7 to 3.8 >> (by creating new venv). > > Great it works. > > Should the documentation be improved to clarify the importance of > re-installing git hooks after upgrading Kallithea or Python? Where and how > would you suggest phrasing it? I did not expect that I need to do this step because I did not install any hooks. So when I got to the step "10. Update Git repository hooks" I ignored it thinking "hey, I do not have any hooks installed". Regards, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: problem with git push
On 21/11/2020 20.11, Thomas De Schampheleire wrote: > Hello, > > El vie., 20 nov. 2020 a las 15:50, Łukasz Michalski () escribió: >> On 20/11/2020 15.42, Mads Kiilerich wrote: >>> On 11/20/20 9:49 AM, Łukasz Michalski wrote: >>>> On 19/11/2020 20.53, Mads Kiilerich wrote: >>>>> On 11/19/20 4:31 PM, Łukasz Michalski wrote: >>>>>> Hi, >>>>>> >>>>>> I have a problem with one of git repos. When I try to push changes I got >>>>>> "remote rejected": >>>>>> >>>>>> [zork@serenity filebench]$ git push zork >>>>>> Password for 'https://zork@': >>>>>> Enumerating objects: 15, done. >>>>>> Counting objects: 100% (15/15), done. >>>>>> Delta compression using up to 8 threads >>>>>> Compressing objects: 100% (8/8), done. >>>>>> Writing objects: 100% (8/8), 3.17 KiB | 1.58 MiB/s, done. >>>>>> Total 8 (delta 7), reused 0 (delta 0), pack-reused 0 >>>>>> To https:// >>>>>>! [remote rejected] branch -> branch (pre-receive hook declined) >>>>>> error: failed to push some refs to 'https://' >>>>>> >>>>>> Kallithea 0.6.2 on Arch Linux. >>>>>> >>>>>> What can I do to resolve this issue? I already tried to increase log >>>>>> level but I cannot find any clues in logs. >>>>> It works for other repos - just not this one? >>>>> >>>>> The pre-receive hook is really not doing anything and should never reject >>>>> pushes. I guess the hooks (for this repo?) isn't installed correctly? >>>>> Inspect and compare hooks/post-receive for working and non-working repos. >>>>> Or try to reinstall all hooks in Admin/Settings/Remap and Rescan/Install >>>>> Git hooks + Overwrite existing Git hooks. >>>>> >>>>> It might be a very fundamental problem (such as finding the right Python >>>>> interpreter to run the hook) so I doubt Kallithea logging could say >>>>> anything. But perhaps the web server error log has something. >>>>> >>>>> /Mads >>>>> >>>> Rescan+Overwrite helped. Thanks! >>>> >>>> I also upgraded kallithea to 0.6.2 changing python version from 3.7 to 3.8 >>>> (by creating new venv). >>> >>> Great it works. >>> >>> Should the documentation be improved to clarify the importance of >>> re-installing git hooks after upgrading Kallithea or Python? Where and how >>> would you suggest phrasing it? >>> >>> I don't know how we can improve this handling. We rely on Git >>> functionality. If you modify hooks/pre-receive to invoke sys.stdin.write or >>> sys.stderr.write or cause an exception (for example by adding an invalid >>> import), you will see that as "remote:" lines on the client side. But if >>> you modify the first line with #! to point at something that doesn't work, >>> Git will be silent - not even a "bad interpreter" message as the shell >>> would do. >>> >>> We could perhaps change the hooks to *always* write something, such as >>> "rejected by Kallithea hook" or "accepted by Kallithea hook" ... and if >>> there is no such message, we can know that the hook failed badly. But that >>> wouldn't be explicit and obvious ... and we did figure it out anyway. >>> >>> /Mads >>> >> I would automate the most common path - python upgrade. >> >> Store python version somewhere in db when hooks are installed. >> Check if hooks needs reinstall on startup. > I don't think that the Python version is the crucial element here. > Rather, any change in the virtualenv path would yield this problem. > Changing the virtualenv path also has impact in other places: it > causes a new path to kallithea-cli, which has impact on e.g. the > authorized_keys file for SSH repository access. > > Having a check on startup like you suggest could perhaps be done: scan > the git hooks, authorized keys, and perhaps other places, to check > whether python and other paths point to the running instance, giving a > warning if it is not the case. > I did not change venv path. When upgrading from 3.7 to 3.8 i removed venv dir and recreated it in the same location. Then I created frontend, default config file and manually merged all changes from old config file to new one. I think that if kallithea installs it's own components in repositories then kallithea should check them on startup and try to reinstall if there is any problem. If reinstall fails then it is time to warn user. I agree that many things can change and break something, but simple python upgrade should work out of the box. Regards, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: problem with git push
On 20/11/2020 15.42, Mads Kiilerich wrote: > On 11/20/20 9:49 AM, Łukasz Michalski wrote: >> On 19/11/2020 20.53, Mads Kiilerich wrote: >>> On 11/19/20 4:31 PM, Łukasz Michalski wrote: >>>> Hi, >>>> >>>> I have a problem with one of git repos. When I try to push changes I got >>>> "remote rejected": >>>> >>>> [zork@serenity filebench]$ git push zork >>>> Password for 'https://zork@': >>>> Enumerating objects: 15, done. >>>> Counting objects: 100% (15/15), done. >>>> Delta compression using up to 8 threads >>>> Compressing objects: 100% (8/8), done. >>>> Writing objects: 100% (8/8), 3.17 KiB | 1.58 MiB/s, done. >>>> Total 8 (delta 7), reused 0 (delta 0), pack-reused 0 >>>> To https:// >>>> ! [remote rejected] branch -> branch (pre-receive hook declined) >>>> error: failed to push some refs to 'https://' >>>> >>>> Kallithea 0.6.2 on Arch Linux. >>>> >>>> What can I do to resolve this issue? I already tried to increase log level >>>> but I cannot find any clues in logs. >>> >>> It works for other repos - just not this one? >>> >>> The pre-receive hook is really not doing anything and should never reject >>> pushes. I guess the hooks (for this repo?) isn't installed correctly? >>> Inspect and compare hooks/post-receive for working and non-working repos. >>> Or try to reinstall all hooks in Admin/Settings/Remap and Rescan/Install >>> Git hooks + Overwrite existing Git hooks. >>> >>> It might be a very fundamental problem (such as finding the right Python >>> interpreter to run the hook) so I doubt Kallithea logging could say >>> anything. But perhaps the web server error log has something. >>> >>> /Mads >>> >> Rescan+Overwrite helped. Thanks! >> >> I also upgraded kallithea to 0.6.2 changing python version from 3.7 to 3.8 >> (by creating new venv). > > > Great it works. > > Should the documentation be improved to clarify the importance of > re-installing git hooks after upgrading Kallithea or Python? Where and how > would you suggest phrasing it? > > I don't know how we can improve this handling. We rely on Git functionality. > If you modify hooks/pre-receive to invoke sys.stdin.write or sys.stderr.write > or cause an exception (for example by adding an invalid import), you will see > that as "remote:" lines on the client side. But if you modify the first line > with #! to point at something that doesn't work, Git will be silent - not > even a "bad interpreter" message as the shell would do. > > We could perhaps change the hooks to *always* write something, such as > "rejected by Kallithea hook" or "accepted by Kallithea hook" ... and if there > is no such message, we can know that the hook failed badly. But that wouldn't > be explicit and obvious ... and we did figure it out anyway. > > /Mads > I would automate the most common path - python upgrade. Store python version somewhere in db when hooks are installed. Check if hooks needs reinstall on startup. (sorry for top-posting in previous reply) Regards, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: problem with git push
I would automate the most common path - python upgrade. Store python version somewhere in db when hooks are installed. Check if hooks needs reinstall on startup. Regards, Łukasz On 20/11/2020 15.42, Mads Kiilerich wrote: > On 11/20/20 9:49 AM, Łukasz Michalski wrote: >> On 19/11/2020 20.53, Mads Kiilerich wrote: >>> On 11/19/20 4:31 PM, Łukasz Michalski wrote: >>>> Hi, >>>> >>>> I have a problem with one of git repos. When I try to push changes I got >>>> "remote rejected": >>>> >>>> [zork@serenity filebench]$ git push zork >>>> Password for 'https://zork@': >>>> Enumerating objects: 15, done. >>>> Counting objects: 100% (15/15), done. >>>> Delta compression using up to 8 threads >>>> Compressing objects: 100% (8/8), done. >>>> Writing objects: 100% (8/8), 3.17 KiB | 1.58 MiB/s, done. >>>> Total 8 (delta 7), reused 0 (delta 0), pack-reused 0 >>>> To https:// >>>> ! [remote rejected] branch -> branch (pre-receive hook declined) >>>> error: failed to push some refs to 'https://' >>>> >>>> Kallithea 0.6.2 on Arch Linux. >>>> >>>> What can I do to resolve this issue? I already tried to increase log level >>>> but I cannot find any clues in logs. >>> >>> It works for other repos - just not this one? >>> >>> The pre-receive hook is really not doing anything and should never reject >>> pushes. I guess the hooks (for this repo?) isn't installed correctly? >>> Inspect and compare hooks/post-receive for working and non-working repos. >>> Or try to reinstall all hooks in Admin/Settings/Remap and Rescan/Install >>> Git hooks + Overwrite existing Git hooks. >>> >>> It might be a very fundamental problem (such as finding the right Python >>> interpreter to run the hook) so I doubt Kallithea logging could say >>> anything. But perhaps the web server error log has something. >>> >>> /Mads >>> >> Rescan+Overwrite helped. Thanks! >> >> I also upgraded kallithea to 0.6.2 changing python version from 3.7 to 3.8 >> (by creating new venv). > > > Great it works. > > Should the documentation be improved to clarify the importance of > re-installing git hooks after upgrading Kallithea or Python? Where and how > would you suggest phrasing it? > > I don't know how we can improve this handling. We rely on Git functionality. > If you modify hooks/pre-receive to invoke sys.stdin.write or sys.stderr.write > or cause an exception (for example by adding an invalid import), you will see > that as "remote:" lines on the client side. But if you modify the first line > with #! to point at something that doesn't work, Git will be silent - not > even a "bad interpreter" message as the shell would do. > > We could perhaps change the hooks to *always* write something, such as > "rejected by Kallithea hook" or "accepted by Kallithea hook" ... and if there > is no such message, we can know that the hook failed badly. But that wouldn't > be explicit and obvious ... and we did figure it out anyway. > > /Mads > ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: problem with git push
On 19/11/2020 20.53, Mads Kiilerich wrote: > On 11/19/20 4:31 PM, Łukasz Michalski wrote: >> Hi, >> >> I have a problem with one of git repos. When I try to push changes I got >> "remote rejected": >> >> [zork@serenity filebench]$ git push zork >> Password for 'https://zork@': >> Enumerating objects: 15, done. >> Counting objects: 100% (15/15), done. >> Delta compression using up to 8 threads >> Compressing objects: 100% (8/8), done. >> Writing objects: 100% (8/8), 3.17 KiB | 1.58 MiB/s, done. >> Total 8 (delta 7), reused 0 (delta 0), pack-reused 0 >> To https:// >> ! [remote rejected] branch -> branch (pre-receive hook declined) >> error: failed to push some refs to 'https://' >> >> Kallithea 0.6.2 on Arch Linux. >> >> What can I do to resolve this issue? I already tried to increase log level >> but I cannot find any clues in logs. > > > It works for other repos - just not this one? > > The pre-receive hook is really not doing anything and should never reject > pushes. I guess the hooks (for this repo?) isn't installed correctly? Inspect > and compare hooks/post-receive for working and non-working repos. Or try to > reinstall all hooks in Admin/Settings/Remap and Rescan/Install Git hooks + > Overwrite existing Git hooks. > > It might be a very fundamental problem (such as finding the right Python > interpreter to run the hook) so I doubt Kallithea logging could say anything. > But perhaps the web server error log has something. > > /Mads > Rescan+Overwrite helped. Thanks! Before discovering this issue I changed repository group for this repo from GroupA to GroupB and then from GroupB to GroupA. I also upgraded kallithea to 0.6.2 changing python version from 3.7 to 3.8 (by creating new venv). Regards, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: memory leak stikes back
On 01/09/2020 09.29, Thomas De Schampheleire wrote: > Hi Łukasz, > > Sorry to have given you confusing information. > > The code snippet I had looked at was: > (kallithea/lib/paster_commands/template.ini.mako) > > %if http_server == 'gearbox': > <%text>## Gearbox default web server ## > use = egg:gearbox#wsgiref > <%text>## nr of worker threads to spawn > threadpool_workers = 1 > <%text>## max request before thread respawn > threadpool_max_requests = 100 > <%text>## option to use threads of process > use_threadpool = true > > [..] > > %elif http_server == 'waitress': > <%text>## WAITRESS ## > use = egg:waitress#main > <%text>## number of worker threads > threads = 1 > <%text>## MAX BODY SIZE 100GB > max_request_body_size = 107374182400 > <%text>## use poll instead of select, fixes fd limits, may not work > on old > <%text>## windows systems. > #asyncore_use_poll = True > > [..] > > %endif > > > The directive 'threadpool_max_requests' that I was referring to is an option > to the 'gearbox' http server, not 'waitress'. Also the 'gunicorn' server has > a similar option set, but it seems that 'waitress' does not support something > directly > (https://docs.pylonsproject.org/projects/waitress/en/stable/arguments.html). > > But 'waitress' is the default setting for kallithea-cli config-create. > > You can try another server using: > kallithea-cli config-create foo.ini http_server=gearbox > > and replace gearbox by the one you would like to choose. Valid options are: > waitress, gearbox, gevent, gunicorn, uwsgi . > > In my own production instance I'm using uwsgi. > > > Mads: I wonder if we should do something about the default 'waitress' setting? > > Thanks, > Thomas > Ok, thanks. Meanwhile I added systemd timer and service to restart kallithea once a week. I think it would be worth doing something about default configuration to prevent memory leaks like this. Do you have such problems with uwsgi configuration? Regards, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: memory leak stikes back
On 29/08/2020 12.26, Thomas De Schampheleire wrote:
> Hi Łukasz,
>
> Apart from some code changes we did to improve/eliminate some caching
> mechanisms, we also added a note to our documentation to clarify the
> expectations. See the bottom of the following page:
> https://kallithea.readthedocs.io/en/stable/overview.html
>
> Basically, we can not avoid certain memory growth in the context of a single
> process, and the recommended approach is to ensure that the web server
> process recreates its workers regularly.
>
> For gearbox/waitress, this can be done automatically with the setting
> threadpool_max_requests, which is set to 100 by default in the ini file
> produced by 'kallithea-cli config-create'.
>
> Is this setting present in your ini file?
> Could you share the ini file?
>
No it is not. I generated new config using kallithea-cli and manually moved
changes from old one.
I attached it below.
I just generated new one and I do not see threadpool_max_requests option in
there.
Regards,
Łukasz
###
###
## Kallithea config file generated with kallithea-cli 0.6.1
##
##
##
## The %(here)s variable will generally be replaced with the parent directory
of ##
## this file. Other use of % must be escaped as %% .
##
###
###
[DEFAULT]
## Email settings ##
####
## Refer to the documentation ("Email settings") for more details.##
####
## It is recommended to use a valid sender address that passes access ##
## validation and spam filtering in mail servers. ##
## 'From' header for application emails. You can optionally add a name.
## Default:
#app_email_from = Kallithea
## Examples:
#app_email_from = Kallithea
#app_email_from = kallithea-nore...@example.com
app_email_from = CodeReview
## Subject prefix for application emails.
## A space between this prefix and the real subject is automatically added.
## Default:
#email_prefix =
## Example:
#email_prefix = [Kallithea]
## Recipients for error emails and fallback recipients of application mails.
## Multiple addresses can be specified, comma-separated.
## Only addresses are allowed, do not add any name part.
## Default:
#email_to =
## Examples:
#email_to = ad...@example.com
#email_to = ad...@example.com,another_ad...@example.com
email_to = r...@domain.pl
## 'From' header for error emails. You can optionally add a name.
## Default: (none)
## Examples:
#error_email_from = Kallithea Errors
#error_email_from = kallithea_err...@example.com
error_email_from = CodeReview
## SMTP server settings
## If specifying credentials, make sure to use secure connections.
## Default: Send unencrypted unauthenticated mails to the specified smtp_server.
## For "SSL", use smtp_use_ssl = true and smtp_port = 465.
## For "STARTTLS", use smtp_use_tls = true and smtp_port = 587.
smtp_server = localhost
smtp_username =
smtp_password =
smtp_port = 25
smtp_use_ssl = false
smtp_use_tls = false
## Entry point for 'gearbox serve'
[server:main]
host = 127.0.0.1
port = 5000
## WAITRESS ##
use = egg:waitress#main
## number of worker threads
threads = 1
## MAX BODY SIZE 100GB
max_request_body_size = 107374182400
## use poll instead of select, fixes fd limits, may not work on old
## windows systems.
#asyncore_use_poll = True
## middleware for hosting the WSGI application under a URL prefix
[filter:proxy-prefix]
use = egg:PasteDeploy#prefix
prefix = /codereview
[app:main]
use = egg:kallithea
## enable proxy prefix middleware
filter-with = proxy-prefix
full_stack = true
static_files = true
## Internationalization (see setup documentation for details)
## By default, the languages requested by the browser are used if available,
with English as default.
## Set i18n.enabled=false to disable automatic language choice.
i18n.enabled = false
## To Force a language, set i18n.enabled=false and specify the language in
i18n.lang.
## Valid values are the names of subdirectories in kallithea/i18n with a
LC_MESSAGES/kallithea.mo
i18n.lang = en
cache_dir = %(here)s/data
index_dir = %(here)s/data/index
## uncomment and set this path to use archive download cache
archive_cache_dir = %(here)s/tarballcache
## change this to unique ID
memory leak stikes back
Hi, Unfortunately after upgrading 0.4.1 to 0.6.1 and python3 gearbox still slowly but surely eats memory. RSS was ~200MB after one day and after a week it grows to 2.1GB. I would like to help resolving this issue but I have no idea where to start. Regards, Łukasz ___ kallithea-general mailing list kallithea-general@sfconservancy.org https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
Re: BUG: Cannot browse files
On 2020-08-23 07:29, Thomas De Schampheleire wrote:
Hi,
On Sat, Aug 22, 2020, 22:49 Łukasz Michalski wrote:
Hi,
I have a problem with a single repository. When I try to use "Files"
tab
I got 500 Internal Server Error (traceback below).
I have 20 repositories and the problem is only with one of them.
Looks
like python3 problem.
[...]
line 587, in __init__
Aug 22 22:29:49 universe gearbox[22393]: self.path =
name.rstrip('/')
Aug 22 22:29:49 universe gearbox[22393]: TypeError: a bytes-like
object
is required, not 'str'
I think this is the problem fixed by following commit on the stable
branch:
https://kallithea-scm.org/repos/kallithea/changeset/f48b12755d83cba1c8e6613e9df2b004ade3721f
There is no release yet with that included, but you could either
install from the repo, or patch locally.
That was it!
Many thanks, I changed both files manually and it works now.
P.S.
Please set Reply-to: kallithea-general@sfconservancy.org for mails sent from
list.
It is unusual that reply is sent to original sender and I have to use "Reply
all".
Regards,
Łukasz
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
