[ https://issues.apache.org/jira/browse/TINKERPOP-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17720641#comment-17720641 ]
Matthew Carr edited comment on TINKERPOP-2943 at 5/8/23 6:34 PM: ----------------------------------------------------------------- I tested your suggestion, and using `long` does indeed resolve the issue. This is a good work-around, but may I suggest that during serialization an integer type is chosen based on the size of the Python `int` which is being serialized. So if the value falls with in the 32-bit signed range it's serialized as it is now, but if it's outside this range and inside the 64-bit signed range, it's serialized as a long. If an `int` falls outside of the 64-bit range, then an exception occurs. Do you think this makes sense? was (Author: JIRAUSER300219): I tested your suggestion, and using `long` does indeed resolve the issue. This is a good work-around, but may I suggest that during serialization an integer type is chosen based on the size of the Python `int` which is being serialized. So if the value falls with in the 32-bit signed range it's serialized as it is now, but if it's outside this range and inside the 64-bit signed range, it's serialized as a long. If an `int` that falls outside of the 64-bit range, then an exception occurs. Do you think this makes sense? > GraphBinary serialization error when vertex ID is larger than 2**31. > -------------------------------------------------------------------- > > Key: TINKERPOP-2943 > URL: https://issues.apache.org/jira/browse/TINKERPOP-2943 > Project: TinkerPop > Issue Type: Bug > Components: python > Affects Versions: 3.6.2 > Environment: Python 3.10 > Reporter: Matthew Carr > Priority: Major > > An exception occurs when using the "graphbinaryV1" serialization format and > an attempt is made to query a vertex whose ID is larger than `2**31`. I > encountered this issue on `gremlinpython` 3.6.2 running on Python 3.10. A > minimal reproduction is as follows: > ``` > from gremlin_python.driver.driver_remote_connection import > DriverRemoteConnection > from gremlin_python.process.anonymous_traversal import traversal > remote = DriverRemoteConnection("ws://127.0.0.1:8182/gremlin", "g") > g = traversal().withRemote(remote) > g.V(2**31 + 1).next() > ``` > This code produces the following stack trace on my machine: > ``` > Traceback (most recent call last): > File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main > return _run_code(code, main_globals, None, > File "/usr/lib/python3.10/runpy.py", line 86, in _run_code > exec(code, run_globals) > File > "/home/matt/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", > line 39, in <module> > cli.main() > File > "/home/matt/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", > line 430, in main > run() > File > "/home/matt/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", > line 284, in run_file > runpy.run_path(target, run_name="__main__") > File > "/home/matt/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", > line 321, in run_path > return _run_module_code(code, init_globals, run_name, > File > "/home/matt/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", > line 135, in _run_module_code > _run_code(code, mod_globals, init_globals, > File > "/home/matt/.vscode/extensions/ms-python.python-2023.4.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", > line 124, in _run_code > exec(code, run_globals) > File "/home/matt/Projects/tada/thermo/repro.py", line 6, in <module> > g.V(2**31 + 1).next() > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/process/traversal.py", > line 117, in next > return self.__next__() > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/process/traversal.py", > line 48, in __next__ > self.traversal_strategies.apply_strategies(self) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/process/traversal.py", > line 684, in apply_strategies > traversal_strategy.apply(traversal) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/remote_connection.py", > line 78, in apply > remote_traversal = self.remote_connection.submit(traversal.bytecode) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/driver_remote_connection.py", > line 104, in submit > result_set = self._client.submit(bytecode, > request_options=self._extract_request_options(bytecode)) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/client.py", > line 150, in submit > return self.submit_async(message, bindings=bindings, > request_options=request_options).result() > File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result > return self.__get_result() > File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in > __get_result > raise self._exception > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/connection.py", > line 73, in cb > f.result() > File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result > return self.__get_result() > File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in > __get_result > raise self._exception > File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run > result = self.fn(*self.args, **self.kwargs) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/protocol.py", > line 86, in write > message = self._message_serializer.serialize_message(request_id, > request_message) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/serializer.py", > line 225, in serialize_message > args = processor_obj.get_op_args(op, args) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/serializer.py", > line 48, in get_op_args > return op_method(args) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/driver/serializer.py", > line 88, in bytecode > args['gremlin'] = self._writer.to_dict(gremlin) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/structure/io/graphbinaryV1.py", > line 168, in to_dict > return self.serializers[t].dictify(obj, self, to_extend) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/structure/io/graphbinaryV1.py", > line 811, in dictify > writer.to_dict(arg, to_extend) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/structure/io/graphbinaryV1.py", > line 168, in to_dict > return self.serializers[t].dictify(obj, self, to_extend) > File > "/home/matt/Projects/tada/thermo/.venv/lib/python3.10/site-packages/gremlin_python/structure/io/graphbinaryV1.py", > line 253, in dictify > to_extend.extend(cls.byte_format_pack(obj)) > struct.error: 'i' format requires -2147483648 <= number <= 2147483647 > ``` > In the actual project where this error was discovered I was passing a vertex > object to the `V()` method, but the underlying issue appears to be the way > that the vertex ID is being serialized. I have a graph with many more than > `2^31` vertices in it, so dealing with IDs larger than this is essential. I > can work around the problem by include the query for the vertex every time, > but this is inefficient. A better solution would be to remove this limitation. > > This may be related to TINKERPOP-2363. -- This message was sent by Atlassian Jira (v8.20.10#820010)