Hi,
> 1. Check that the client is properly reading the whole of the response
> 9even if zero bytes) and is actually closing the connection, or
> returning it to the connection pool. Check by running "netstat" to see
> TCp connections ("-t" on *nix)
With netstat I saw several connections in TIME_WAIT state when running my test.
I think it means that the tcp-connections have been properly closed.
I understand that unproperly terminated tcp-connections could lead to error
case "Maximum lock count exceeded", but what could cause jena-fuseki 5.1.0 use
much more memory than 3.14 did with exactly same program code and input in the
client side and same data in the database ?
I also collected the return values of queries and updates and noticed that,
they should be fine as well:
i.e. the following code
def run_update(query_formatted):
sparql = init_rdf_triple_store_for('update')
sparql.method = 'POST'
sparql.setQuery(fix_eol_chars(query_formatted))
ret = sparql.query()
print("run_update, ret:", ret)
sparql = None
def init_rdf_triple_store_for(what):
curframe = inspect.currentframe()
calframe = inspect.getouterframes(curframe, 2)
try:
endpoint = settings.JENADATABASE.format(JENA_SERVER_URL, what)
sparql = SPARQLWrapper(endpoint)
sparql.setHTTPAuth(BASIC)
sparql.setCredentials(JENA_SERVER_USERNAME, JENA_SERVER_PASSWORD)
return sparql
except BaseException:
return ''
returned
run_update, ret: <SPARQLWrapper.Wrapper.QueryResult object at
0x00007F15172B68B0>
{"requestedFormat" : 'xml',
"response (a file-like object, as return by the urllib2.urlopen library call)"
: {
"url" : "http://localhost:3030/ds/update";,
"code" : "200",
"headers" : Date: Tue, 29 Oct 2024 10:05:21 GMT
Vary: Accept-Encoding
Vary: Origin
Set-Cookie: JSESSIONID=node01v3s1dzzrrzzu1g8mi8ek6acvj75760.node0; Path=/
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/; Max-Age=0; Expires=Mon, 28-Oct-2024
10:05:21 GMT; SameSite=lax
Fuseki-Request-Id: 110110
Content-Type: text/plain;charset=utf-8
Content-Length: 17
Connection: close
}}
Of course I could run the whole test and check the return-code (and Connection
close or not ) for each update, but I don't think it would help. (The above was
produced by running just few lines of the huge excel-file).
Br, Jaana
> 28.10.2024 19.59 EET Andy Seaborne <[email protected]> kirjoitti:
>
>
> On 28/10/2024 10:42, [email protected] wrote:
> > Hello, thanks for your answer.
> >
> >> I don't see why you said UI doesn't work.
> > It was a mistake, I'm sorry. It works, but currentky I have problems with
> > version 5.1.0.
> >
> > Let me explain:
> > I have been running image docker.io/stain/jena-fuseki:3.14.0 for years
> > without problems, but after upgrade to 5.x I started to get problems.
> >
> > I pulled docker.io/stain/jena-fuseki:5.1.0 from docker hub, deployed into
> > my test server and started to test. I tried to update my
> > jena-fuseki-database running in podman container by reading an excel-file
> > line by line in Python-code and running 6 sparql updates for each line. My
> > excel-file has 21 412 lines.
> >
> > With 3.14.0-version this update passed in 4 and half hours, but with 5.1.0
> > it does not pass at all; either the execution stops for running out of
> > memory or it stops for error message “Maximum lock count exceeded”.
> >
> > In 5.1.0 the execution also slows down: In the beginning about 50
> > rows/minute become handled but after 5 hours only 5 rows/minute become
> > handled. I guess this is due to running short of memory.
> >
> > I also guess that there are no memory leaks, but the carbage collection
> > should be configured in different manner. We have just the following
> > settings in our container run command:
> > --memory 6g \
> > --env="JVM_ARGS"="-Xms6144m
> > -Xmx6144m" \
> >
> > Maybe I should have something else, too. What ?
>
> 1. Check that the client is properly reading the whole of the response
> 9even if zero bytes) and is actually closing the connection, or
> returning it to the connection pool. Check by running "netstat" to see
> TCp connections ("-t" on *nix)
>
> 2. On Windows - don't run client and server on the same machine; Windows
> is slow to recycle TCP connections under load.
>
> >
> > Thanks so much if you could help !
> > Br, Jaana
> >
> >
> >> 28.10.2024 12.00 EET Lorenz Buehmann <[email protected]>
> >> kirjoitti:
> >>
> >>
> >> The advice was to simply try out version 5.1.0 from the
> >> stain/jena-fuseki repo: https://hub.docker.com/r/stain/jena-fuseki/tags
> >>
> >> The image does nothing more than downloading Fuseki dist tarball,
> >> setting all things up and starting the server. I don't see why you said
> >> UI doesn't work.
> >>
> >>
> >> Lorenz
> >>
> >> On 28.10.24 06:09, [email protected] wrote:
> >>> Hello, you write below, that stain/* shouldn't be used as it is not part
> >>> of Apache Jena project and that there's 5.1.0 in dockerhub. Yes, there
> >>> are several 5.* images in dockerhub, but I don't know which ones of them
> >>> are part of Apache Jena project. Can you help ?
> >>>
> >>> Jaana
> >>>
> >>>
> >>>> 17.10.2024 14.38 EEST Andy Seaborne <[email protected]> kirjoitti:
> >>>>
> >>>>
> >>>> On 17/10/2024 10:31, [email protected] wrote:
> >>>>>> You should upgrade to Jena 5.x to get security updates.
> >>>>> The problem is that UI does not work in stain/jena-fuseki-version
> >>>>> higher than 4.0.0.
> >>>> 1. stain/* is not a product of the Apache Jena project.
> >>>>
> >>>> 2. Are you sure? There is a 5.1.0 on dockerhub.
> >>>>
> >>>> Andy
> >>>>
> >>>>> Jaana
> >>>>>
> >>>>>> 17.10.2024 11.45 EEST Andy Seaborne <[email protected]> kirjoitti:
> >>>>>>
> >>>>>>
> >>>>>> On 17/10/2024 03:37, [email protected] wrote:
> >>>>>>>
> >>>>>>> I'm running docker.io/stain/jena-fuseki:3.14.0 in podman container on
> >>>>>>> linux redHat host. I a very heavy load it can happen that
> >>>>>>> /var/log/messages-file receives huge amount of INFO-level messages
> >>>>>>> from jena-fuseki. How can I change the log level of my container to
> >>>>>>> get rid of these messages ?
> >>>>>>>
> >>>>>>> This is similar with
> >>>>>>> https://stackoverflow.com/questions/71615972/change-jena-fuseki-logging-level
> >>>>>>>
> >>>>>>> https://stackoverflow.com/questions/71615972/change-jena-fuseki-logging-level
> >>>>>>> , but the problem now is that I cannot find the file
> >>>>>>> log4j2.properties from inside the container.
> >>>>>>>
> >>>>>>> Br, Jaana
> >>>>>> 3.14.0 used log4j1 The file will be called log4j.properties.
> >>>>>>
> >>>>>> If it does not exist, then the server uses a built in default. You can
> >>>>>> place a logging setup in the container at the expected pl;ace and it
> >>>>>> should use on start-up.
> >>>>>>
> >>>>>> You should upgrade to Jena 5.x to get security updates.
> >>>>>>
> >>>>>> Andy
> >>
> >> --
> >> Lorenz Bühmann
> >> Research Associate/Scientific Developer
> >>
> >> Email [email protected]
> >>
> >> Institute for Applied Informatics e.V. (InfAI) | Goerdelerring 9 | 04109
> >> Leipzig | Germany
