Hi Sean, Would you be willing to send us a Pull Request with the "local change to scripts/test-rest.ts to handle the chunking"? https://github.com/DSpace/dspace-angular/
That'd be extremely helpful in us better understanding this issue & perhaps debugging whether the UI can currently handle chunked responses (and if not, finding a way to fix it). At the very least, it sounds like we have a bug in the "test-rest.ts" script (the code behind yarn test:rest). Thanks, Tim ________________________________ From: [email protected] <[email protected]> on behalf of [email protected] <[email protected]> Sent: Friday, September 9, 2022 3:15 PM To: DSpace Technical Support <[email protected]> Subject: [dspace-tech] Re: ERROR: INVALID DSPACE REST API! Response is not valid JSON! I just ran into this as well with DSpace 7.3. We have an extra layer in our environment with both nginx and an F5 load balancer acting as reverse proxies. This is definitely a "Transfer-Encoding: chunked" issue as Tim alluded to in the other discussion thread he linked below. The "yarn test:rest" test does not handle the chunked data responses leading to the error. A local change to scripts/test-rest.ts to handle the chunking resolved the issue for the test. However, I did some more testing (without the local fix) to learn about the chunking happening at the different layers in our environment (F5 load balancer -> nginx reverse proxy -> Tomcat 9 backend). A direct "yarn test:rest" against Tomcat 9.0.62 worked fine, although HTTP 1.1 is used by default and the response had "Transfer-Encoding: chunked". In this case the chunk size defined by Tomcat 9 was large enough to contain the complete JSON response and there was no validation error. A direct "yarn test:rest" against our nginx reverse proxy produced the JSON validation error message, and it appeared there were about two chunks (based on whatever default size nginx 1.20.1 uses). In this case, setting "chunked_transfer_encoding off;" in the nginx config and restarting nginx resolved the issue since this disables nginx chunking even for HTTP 1.1. The final test against our F5 load balancer showed that it was chunking the data even further (resulting in about seven chunks rather than nginx's two chunks). This occurs even if chunking was turned off for nginx. I have not yet followed through with our central IT to adjust our F5, but I suspect that disabling chunking there could be done as well. Regardless, if this is only an issue with "yarn test:rest" then no big deal. However, if other parts of DSpace 7.3 are not handling chunked transfer encoding properly, then that could lead to a variety of errors depending on the environment. I would certainly prefer to avoid disabling chunking at all proxy layers so I'm hoping this is just an issue with the test case. -- Sean On Friday, 27 May 2022 at 12:10:57 UTC-5 [email protected] wrote: Thanks a lot Tim. If I find any error along the way, I will make sure to report it. Diego. On Friday, May 27, 2022 at 11:28:25 AM UTC-5 Tim Donohue wrote: Hi Diego, I've seen this one other time, but the user who encountered it never responded on what may have caused it. See this thread https://groups.google.com/g/dspace-tech/c/S3yjduTP9Xg/m/Nv5-v6ooBAAJ My suspicion is that some Proxy settings or firewall settings might simply be confusing the `yarn test:rest` script. I've not been able to find a way to reproduce this error though, so it's extremely hard to debug. All that said, if everything in the UI is working for you (and it sounds like it may be), then I'd say there's no problem, and the output of the `yarn test:rest` script can be ignored in this scenario. It's very possible that this is a bug in the `yarn test:rest` script -- and there may be scenarios where this script is creating a false positive error message. (Again though, it's been very hard to debug this, as I've not yet been able to find a developer who has been able to reproduce this problem) Tim On Wednesday, May 25, 2022 at 6:35:35 AM UTC-5 [email protected] wrote: Hi, so ive installed DSpace 7.2 behind nginx #NGINX CONFIG ... server { listen 443 ssl; server_name repositorio.emgs.mx<http://repositorio.emgs.mx>; ssl_certificate /etc/nginx/ssl/emgs.cer; ssl_certificate_key /etc/nginx/ssl/emgs.key; ssl_session_cache builtin:1000 shared:SSL:10m; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4; ssl_prefer_server_ciphers on; location /server { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://repositorio.emgs.mx:8080/server; proxy_redirect http://repositorio.emgs.mx:8080/server https://repositorio.emgs.mx/server; } location / { proxy_pass http://repositorio.emgs.mx:4000; proxy_read_timeout 90; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect http://repositorio.emgs.mx:4000 https://repositorio.emgs.mx; } } #BACKEND local.cfg dspace.server.url = https://repositorio.emgs.mx/server dspace.ui.url = https://repositorio.emgs.mx solr.server = http://repositorio.emgs.mx:8983/solr #FRONTEND config.prod.yml ui: ssl: false host: repositorio.emgs.mx<http://repositorio.emgs.mx> port: 4000 nameSpace: / rest: ssl: true host: repositorio.emgs.mx<http://repositorio.emgs.mx> port: 443 nameSpace: /server #TOMCAT <Connector port="8080" minSpareThreads="25" enableLookups="false" connectionTimeout="20000" disableUploadTimeout="true" URIEncoding="UTF-8"/> Everything seems to be running fine. Can log in, REST API appears to be working fine, but th yarn test gives me an error message as it is being chunked up into pieces. Read all troubleshooting but ive checked all my configs and everything seems to be fine. yarn test:rest yarn run v1.22.18 $ ts-node --project ./tsconfig.ts-node.json scripts/test-rest.ts Building production app config Overriding app config with /home/dspace/dspace-front-source/config/config.prod.yml ...Testing connection to REST API at https://repositorio.emgs.mx/server/api... RESPONSE: 200 ERROR: INVALID DSPACE REST API! Response is not valid JSON! Response returned: {"dspaceUI":"https://repositorio.emgs.mx","dspaceName":"Repositorio Sanidad Militar","dspaceServer":"https://repositorio.emgs.mx/server","dspaceVersion":"DSpace 7.2.1","type":"root","_links":{"authn":{"href":"https://repositorio.emgs.mx/server/api/authn"},"authorizations":{"href":"https://repositorio.emgs.mx/server/api/authz/authorizations"},"bitstreamformats":{"href":"https://repositorio.emgs.mx/server/api/core/bitstreamformats"},"bitstreams":{"href":"https://repositorio.emgs.mx/server/api/core/bitstreams"},"browses":{"href":"https://repositorio.emgs.mx/server/api/discover/browses"},"bundles":{"href":"https://repositorio.emgs.mx/server/api/core/bundles"},"claimedtasks":{"href":"https://repositorio.emgs.mx/server/api/workflow/claimedtasks"},"claimedtasks-search":{"href":"https://repositorio.emgs.mx/server/api/workflow/claimedtasks/search"},"collections":{"href":"https://repositorio.emgs.mx/server/api/core/collections"},"communities":{"href":"https://repositorio.emgs.mx/server/api/core/communities"},"discover":{"href":"https://repositorio.emgs.mx/server/api/discover"},"dso":{"href":"https://repositorio.emgs.mx/server/api/dso/find{?uuid}<https://repositorio.emgs.mx/server/api/dso/find%7B?uuid%7D>","templated":true},"entitytypes":{"href":"https://repositorio.emgs.mx/server/api/core/entitytypes"},"eperson-registration":{"href":"https://repositorio.emgs.mx/server/api/eperson/registrations"},"epersons":{"href":"https://repositorio.emgs.mx/server/api/eperson/epersons"},"externalsources":{"href":"https://repositorio.emgs.mx/server/api/integration/externalsources"},"features":{"href":"https://repositorio.emgs.mx/server/api/authz/features"},"feedbacks":{"href":"https://repositorio.emgs.mx/server/api/tools/feedbacks"},"groups":{"href":"https://repositorio.emgs.mx/server/api/eperson/groups"},"itemrequests":{"href":"https://repositorio.emgs.mx/server/api/tools/itemrequests"},"items":{"href":"https://repositorio.emgs.mx/server/api/core/items"},"itemtemplates":{"href":"https://repositorio.emgs.mx/server/api/core/itemtemplates"},"metadatafields":{"href":"https://repositorio.emgs.mx/server/api/core/metadatafields"},"metadataschemas":{"href":"https://repositorio.emgs.mx/server/api/core/metadataschemas"},"oidc":{"href":"https://repositorio.emgs.mx/server/api/authn"},"pid":{"href":"https://repositorio.emgs.mx/server/api/pid/find{?id}<https://repositorio.emgs.mx/server/api/pid/find%7B?id%7D>","templated":true},"pooltasks":{"href":"https://repositorio.emgs.mx/server/api/workflow/pooltasks"},"pooltasks-search":{"href":"https://repositorio.emgs.mx/server/api/workflow/pooltasks/search"},"processes":{"href":"https://repositorio.emgs.mx/server/api/system/processes"},"properties":{"href":"https://repositorio.emgs.mx/server/api/config/properties"},"registrations":{"href":"https://repositorio.emgs.mx/server/api/eperson/registrations"},"relationships":{"href":"https://repositorio.emgs.mx/server/api/core/relationships"},"relationshiptypes":{"href":"https://repositorio.emgs.mx/server/api/core/relationshiptypes"},"resourcepolicies":{"href":"https://repositorio.emgs.mx/server/api/authz/resourcepolicies"},"resourcepolicies-search":{"href":"https://repositorio.emgs.mx/server/api/authz/resourcepolicies/search"},"scripts":{"href":"https://repositorio.emgs.mx/server/api/system/scripts"},"self":{"href":"https://repositorio.emgs.mx/server/api"},"sites":{"href":"https://repositorio.emgs.mx/server/api/core/sites"},"statistics":{"href":"https://repositorio.emgs.mx/server/api/statistics"},"submissionaccessoptions":{"href":"https://repositor ERROR: INVALID DSPACE REST API! Response is not valid JSON! Response returned: io.emgs.mx/server/api/config/submissionaccessoptions<http://io.emgs.mx/server/api/config/submissionaccessoptions>"},"submissioncclicenseUrls":{"href":"https://repositorio.emgs.mx/server/api/config/submissioncclicenseUrls"},"submissioncclicenseUrls-search":{"href":"https://repositorio.emgs.mx/server/api/config/submissioncclicenseUrls/search"},"submissioncclicenses":{"href":"https://repositorio.emgs.mx/server/api/config/submissioncclicenses"},"submissiondefinitions":{"href":"https://repositorio.emgs.mx/server/api/config/submissiondefinitions"},"submissionforms":{"href":"https://repositorio.emgs.mx/server/api/config/submissionforms"},"submissionsections":{"href":"https://repositorio.emgs.mx/server/api/config/submissionsections"},"submissionuploads":{"href":"https://repositorio.emgs.mx/server/api/config/submissionuploads"},"usagereports":{"href":"https://repositorio.emgs.mx/server/api/statistics/usagereports"},"versionhistories":{"href":"https://repositorio.emgs.mx/server/api/versioning/versionhistories"},"versions":{"href":"https://repositorio.emgs.mx/server/api/versioning/versions"},"vocabularies":{"href":"https://repositorio.emgs.mx/server/api/submission/vocabularies"},"vocabularyEntryDetails":{"href":"https://repositorio.emgs.mx/server/api/submission/vocabularyEntryDetails"},"vocabularyEntryDetails-search":{"href":"https://repositorio.emgs.mx/server/api/submission/vocabularyEntryDetails/search"},"workflowactions":{"href":"https://repositorio.emgs.mx/server/api/config/workflowactions"},"workflowdefinitions":{"href":"https://repositorio.emgs.mx/server/api/config/workflowdefinitions"},"workflowitems":{"href":"https://repositorio.emgs.mx/server/api/workflow/workflowitems"},"workflowsteps":{"href":"https://repositorio.emgs.mx/server/api/config/workflowsteps"},"workspaceitems":{"href":"https://repositorio.emgs.mx/server/api/submission/workspaceitems"}}} Done in 4.62s. I would really appreciate the help. Diego, -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to a topic in the Google Groups "DSpace Technical Support" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/dspace-tech/XU7KFZGOr94/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected]<mailto:[email protected]>. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/0feb7379-d24a-4b5d-94d8-134b2e4a7bc0n%40googlegroups.com<https://groups.google.com/d/msgid/dspace-tech/0feb7379-d24a-4b5d-94d8-134b2e4a7bc0n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/PH0PR22MB3274B9A52EF28305FA5CFF92ED449%40PH0PR22MB3274.namprd22.prod.outlook.com.
