I just ran into this as well with DSpace 7.3. We have an extra layer in our environment with both nginx and an F5 load balancer acting as reverse proxies.
This is definitely a "Transfer-Encoding: chunked" issue as Tim alluded to in the other discussion thread he linked below. The "yarn test:rest" test does not handle the chunked data responses leading to the error. A local change to scripts/test-rest.ts to handle the chunking resolved the issue for the test. However, I did some more testing (without the local fix) to learn about the chunking happening at the different layers in our environment (F5 load balancer -> nginx reverse proxy -> Tomcat 9 backend). A direct "yarn test:rest" against Tomcat 9.0.62 worked fine, although HTTP 1.1 is used by default and the response had "Transfer-Encoding: chunked". In this case the chunk size defined by Tomcat 9 was large enough to contain the complete JSON response and there was no validation error. A direct "yarn test:rest" against our nginx reverse proxy produced the JSON validation error message, and it appeared there were about two chunks (based on whatever default size nginx 1.20.1 uses). In this case, setting "chunked_transfer_encoding off;" in the nginx config and restarting nginx resolved the issue since this disables nginx chunking even for HTTP 1.1. The final test against our F5 load balancer showed that it was chunking the data even further (resulting in about seven chunks rather than nginx's two chunks). This occurs even if chunking was turned off for nginx. I have not yet followed through with our central IT to adjust our F5, but I suspect that disabling chunking there could be done as well. Regardless, if this is only an issue with "yarn test:rest" then no big deal. However, if other parts of DSpace 7.3 are not handling chunked transfer encoding properly, then that could lead to a variety of errors depending on the environment. I would certainly prefer to avoid disabling chunking at all proxy layers so I'm hoping this is just an issue with the test case. -- Sean On Friday, 27 May 2022 at 12:10:57 UTC-5 [email protected] wrote: > Thanks a lot Tim. If I find any error along the way, I will make sure to > report it. > > Diego. > > On Friday, May 27, 2022 at 11:28:25 AM UTC-5 Tim Donohue wrote: > >> Hi Diego, >> >> I've seen this one other time, but the user who encountered it never >> responded on what may have caused it. See this thread >> https://groups.google.com/g/dspace-tech/c/S3yjduTP9Xg/m/Nv5-v6ooBAAJ >> >> My suspicion is that some Proxy settings or firewall settings might >> simply be confusing the `yarn test:rest` script. I've not been able to >> find a way to reproduce this error though, so it's extremely hard to debug. >> >> All that said, if everything in the UI is working for you (and it sounds >> like it may be), then I'd say there's no problem, and the output of the >> `yarn test:rest` script can be ignored in this scenario. It's very >> possible that this is a bug in the `yarn test:rest` script -- and there may >> be scenarios where this script is creating a false positive error message. >> (Again though, it's been very hard to debug this, as I've not yet been able >> to find a developer who has been able to reproduce this problem) >> >> Tim >> >> On Wednesday, May 25, 2022 at 6:35:35 AM UTC-5 [email protected] wrote: >> >>> Hi, so ive installed DSpace 7.2 behind nginx >>> >>> #NGINX CONFIG >>> ... >>> server { >>> listen 443 ssl; >>> server_name repositorio.emgs.mx; >>> ssl_certificate /etc/nginx/ssl/emgs.cer; >>> ssl_certificate_key /etc/nginx/ssl/emgs.key; >>> ssl_session_cache builtin:1000 shared:SSL:10m; >>> ssl_protocols TLSv1 TLSv1.1 TLSv1.2; >>> ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4; >>> ssl_prefer_server_ciphers on; >>> >>> location /server { >>> proxy_set_header Host $host; >>> proxy_set_header X-Real-IP $remote_addr; >>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; >>> proxy_set_header X-Forwarded-Proto $scheme; >>> proxy_pass http://repositorio.emgs.mx:8080/server; >>> proxy_redirect http://repositorio.emgs.mx:8080/server >>> https://repositorio.emgs.mx/server; >>> } >>> >>> location / { >>> proxy_pass http://repositorio.emgs.mx:4000; >>> proxy_read_timeout 90; >>> proxy_set_header X-Real-IP $remote_addr; >>> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; >>> proxy_set_header X-Forwarded-Proto $scheme; >>> proxy_redirect http://repositorio.emgs.mx:4000 >>> https://repositorio.emgs.mx; >>> } >>> } >>> >>> #BACKEND local.cfg >>> dspace.server.url = https://repositorio.emgs.mx/server >>> dspace.ui.url = https://repositorio.emgs.mx >>> solr.server = http://repositorio.emgs.mx:8983/solr >>> >>> >>> #FRONTEND config.prod.yml >>> ui: >>> ssl: false >>> host: repositorio.emgs.mx >>> port: 4000 >>> nameSpace: / >>> >>> rest: >>> ssl: true >>> host: repositorio.emgs.mx >>> port: 443 >>> nameSpace: /server >>> >>> >>> #TOMCAT >>> <Connector port="8080" >>> minSpareThreads="25" >>> enableLookups="false" >>> connectionTimeout="20000" >>> disableUploadTimeout="true" >>> URIEncoding="UTF-8"/> >>> >>> >>> >>> >>> Everything seems to be running fine. Can log in, REST API appears to be >>> working fine, but th yarn test gives me an error message as it is being >>> chunked up into pieces. Read all troubleshooting but ive checked all my >>> configs and everything seems to be fine. >>> >>> yarn test:rest >>> yarn run v1.22.18 >>> $ ts-node --project ./tsconfig.ts-node.json scripts/test-rest.ts >>> Building production app config >>> Overriding app config with >>> /home/dspace/dspace-front-source/config/config.prod.yml >>> ...Testing connection to REST API at >>> https://repositorio.emgs.mx/server/api... >>> >>> RESPONSE: 200 >>> >>> ERROR: INVALID DSPACE REST API! Response is not valid JSON! >>> Response returned: >>> {"dspaceUI":"https://repositorio.emgs.mx","dspaceName":"Repositorio >>> Sanidad >>> Militar","dspaceServer":"https://repositorio.emgs.mx/server","dspaceVersion":"DSpace >>> >>> 7.2.1","type":"root","_links":{"authn":{"href":" >>> https://repositorio.emgs.mx/server/api/authn >>> "},"authorizations":{"href":" >>> https://repositorio.emgs.mx/server/api/authz/authorizations >>> "},"bitstreamformats":{"href":" >>> https://repositorio.emgs.mx/server/api/core/bitstreamformats >>> "},"bitstreams":{"href":" >>> https://repositorio.emgs.mx/server/api/core/bitstreams >>> "},"browses":{"href":" >>> https://repositorio.emgs.mx/server/api/discover/browses >>> "},"bundles":{"href":" >>> https://repositorio.emgs.mx/server/api/core/bundles >>> "},"claimedtasks":{"href":" >>> https://repositorio.emgs.mx/server/api/workflow/claimedtasks >>> "},"claimedtasks-search":{"href":" >>> https://repositorio.emgs.mx/server/api/workflow/claimedtasks/search >>> "},"collections":{"href":" >>> https://repositorio.emgs.mx/server/api/core/collections >>> "},"communities":{"href":" >>> https://repositorio.emgs.mx/server/api/core/communities >>> "},"discover":{"href":"https://repositorio.emgs.mx/server/api/discover >>> "},"dso":{"href":"https://repositorio.emgs.mx/server/api/dso/find{?uuid} >>> ","templated":true},"entitytypes":{"href":" >>> https://repositorio.emgs.mx/server/api/core/entitytypes >>> "},"eperson-registration":{"href":" >>> https://repositorio.emgs.mx/server/api/eperson/registrations >>> "},"epersons":{"href":" >>> https://repositorio.emgs.mx/server/api/eperson/epersons >>> "},"externalsources":{"href":" >>> https://repositorio.emgs.mx/server/api/integration/externalsources >>> "},"features":{"href":" >>> https://repositorio.emgs.mx/server/api/authz/features >>> "},"feedbacks":{"href":" >>> https://repositorio.emgs.mx/server/api/tools/feedbacks >>> "},"groups":{"href":" >>> https://repositorio.emgs.mx/server/api/eperson/groups >>> "},"itemrequests":{"href":" >>> https://repositorio.emgs.mx/server/api/tools/itemrequests >>> "},"items":{"href":"https://repositorio.emgs.mx/server/api/core/items >>> "},"itemtemplates":{"href":" >>> https://repositorio.emgs.mx/server/api/core/itemtemplates >>> "},"metadatafields":{"href":" >>> https://repositorio.emgs.mx/server/api/core/metadatafields >>> "},"metadataschemas":{"href":" >>> https://repositorio.emgs.mx/server/api/core/metadataschemas >>> "},"oidc":{"href":"https://repositorio.emgs.mx/server/api/authn >>> "},"pid":{"href":"https://repositorio.emgs.mx/server/api/pid/find{?id} >>> ","templated":true},"pooltasks":{"href":" >>> https://repositorio.emgs.mx/server/api/workflow/pooltasks >>> "},"pooltasks-search":{"href":" >>> https://repositorio.emgs.mx/server/api/workflow/pooltasks/search >>> "},"processes":{"href":" >>> https://repositorio.emgs.mx/server/api/system/processes >>> "},"properties":{"href":" >>> https://repositorio.emgs.mx/server/api/config/properties >>> "},"registrations":{"href":" >>> https://repositorio.emgs.mx/server/api/eperson/registrations >>> "},"relationships":{"href":" >>> https://repositorio.emgs.mx/server/api/core/relationships >>> "},"relationshiptypes":{"href":" >>> https://repositorio.emgs.mx/server/api/core/relationshiptypes >>> "},"resourcepolicies":{"href":" >>> https://repositorio.emgs.mx/server/api/authz/resourcepolicies >>> "},"resourcepolicies-search":{"href":" >>> https://repositorio.emgs.mx/server/api/authz/resourcepolicies/search >>> "},"scripts":{"href":" >>> https://repositorio.emgs.mx/server/api/system/scripts"},"self":{"href":"; >>> https://repositorio.emgs.mx/server/api"},"sites":{"href":"; >>> https://repositorio.emgs.mx/server/api/core/sites >>> "},"statistics":{"href":" >>> https://repositorio.emgs.mx/server/api/statistics >>> "},"submissionaccessoptions":{"href":"https://repositor >>> ERROR: INVALID DSPACE REST API! Response is not valid JSON! >>> Response returned: >>> io.emgs.mx/server/api/config/submissionaccessoptions >>> "},"submissioncclicenseUrls":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissioncclicenseUrls >>> "},"submissioncclicenseUrls-search":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissioncclicenseUrls/search >>> "},"submissioncclicenses":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissioncclicenses >>> "},"submissiondefinitions":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissiondefinitions >>> "},"submissionforms":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissionforms >>> "},"submissionsections":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissionsections >>> "},"submissionuploads":{"href":" >>> https://repositorio.emgs.mx/server/api/config/submissionuploads >>> "},"usagereports":{"href":" >>> https://repositorio.emgs.mx/server/api/statistics/usagereports >>> "},"versionhistories":{"href":" >>> https://repositorio.emgs.mx/server/api/versioning/versionhistories >>> "},"versions":{"href":" >>> https://repositorio.emgs.mx/server/api/versioning/versions >>> "},"vocabularies":{"href":" >>> https://repositorio.emgs.mx/server/api/submission/vocabularies >>> "},"vocabularyEntryDetails":{"href":" >>> https://repositorio.emgs.mx/server/api/submission/vocabularyEntryDetails >>> "},"vocabularyEntryDetails-search":{"href":" >>> https://repositorio.emgs.mx/server/api/submission/vocabularyEntryDetails/search >>> "},"workflowactions":{"href":" >>> https://repositorio.emgs.mx/server/api/config/workflowactions >>> "},"workflowdefinitions":{"href":" >>> https://repositorio.emgs.mx/server/api/config/workflowdefinitions >>> "},"workflowitems":{"href":" >>> https://repositorio.emgs.mx/server/api/workflow/workflowitems >>> "},"workflowsteps":{"href":" >>> https://repositorio.emgs.mx/server/api/config/workflowsteps >>> "},"workspaceitems":{"href":" >>> https://repositorio.emgs.mx/server/api/submission/workspaceitems"}}} >>> Done in 4.62s. >>> >>> >>> I would really appreciate the help. >>> >>> Diego, >>> >>> -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/0feb7379-d24a-4b5d-94d8-134b2e4a7bc0n%40googlegroups.com.
