[grpc-io] Re: Cannot generate Python modules from containerd's API .proto files

[TheDiveO]

After a deeper analysis and some trial+error I have now a working 
containerd Python API package. Bottom line was to comb through the process 
of generating a proper .proto files file structure of only those including 
vendored dependencies needed and then fixing the import paths to avoid any 
import path aliasing. The details can be found here: 
https://github.com/TheDiveO/pycontainerd/blob/master/script/genpb2.sh ... 
heavens, that containerd API and vendor tree really is ... challenging ... 
when generating Python modules.

As I can work with the unmodified existing containerd sources, I can now 
send a PR upstream to the Python containerd package project.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/ffa453d3-9291-444a-ae9c-9144d7070a18n%40googlegroups.com.

[grpc-io] Re: Cannot generate Python modules from containerd's API .proto files

[TheDiveO]

This is the error happening when trying to use the created packages on 
Ubuntu 20.10 on ARM64; the lsctr.py script is from an example script from 
the Github project https://github.com/siemens/pycontainerd, importing and 
using the generated Python3 modules:

File "lsctr.py", line 28, in 
from containerd.services.namespaces.v1 import namespace_pb2_grpc, 
namespace_pb2
  File 
"/home/x/.local/lib/python3.8/site-packages/containerd/services/namespaces/v1/namespace_pb2_grpc.py",
 
line 4, in 
from containerd.services.namespaces.v1 import namespace_pb2 as 
containerd_dot_services_dot_namespaces_dot_v1_dot_namespace__pb2
  File 
"/home/x/.local/lib/python3.8/site-packages/containerd/services/namespaces/v1/namespace_pb2.py",
 
line 16, in 
from containerd.vendor.gogoproto import gogo_pb2 as 
containerd_dot_vendor_dot_gogoproto_dot_gogo__pb2
  File 
"/home/x/.local/lib/python3.8/site-packages/containerd/vendor/gogoproto/gogo_pb2.py",
 
line 16, in 
from containerd.vendor.google.protobuf import descriptor_pb2 as 
containerd_dot_vendor_dot_google_dot_protobuf_dot_descriptor__pb2
  File 
"/home/x/.local/lib/python3.8/site-packages/containerd/vendor/google/protobuf/descriptor_pb2.py",
 
line 2005, in 
'Location' : _reflection.GeneratedProtocolMessageType('Location', 
(_message.Message,), {
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/internal/python_message.py",
 
line 196, in __init__
_AttachFieldHelpers(cls, field)
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/internal/python_message.py",
 
line 303, in _AttachFieldHelpers
field_descriptor.GetOptions().packed)
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor.py", 
line 167, in GetOptions
from google.protobuf import descriptor_pb2
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor_pb2.py", 
line 130, in 
_sym_db.RegisterEnumDescriptor(_FIELDDESCRIPTORPROTO_TYPE)
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/symbol_database.py",
 
line 108, in RegisterEnumDescriptor
self.pool._AddEnumDescriptor(enum_descriptor)
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor_pool.py",
 
line 263, in _AddEnumDescriptor
self._CheckConflictRegister(enum_desc, enum_desc.full_name, file_name)
  File 
"/home/x/.local/lib/python3.8/site-packages/google/protobuf/descriptor_pool.py",
 
line 191, in _CheckConflictRegister
raise TypeError(error_msg)
TypeError: Conflict register for file "google/protobuf/descriptor.proto": 
google.protobuf.FieldDescriptorProto.Type is already defined in file 
"containerd/vendor/google/protobuf/descriptor.proto". Please fix the 
conflict by adding package name on the proto file, or use different name 
for the duplication.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/bfd992b6-ff45-4de9-8be1-5ec83e1288b9n%40googlegroups.com.

[grpc-io] Re: Cannot generate Python modules from containerd's API .proto files

['Richard Belleville' via grpc.io]

Can you please include the runtime errors you're encountering?
On Wednesday, December 9, 2020 at 11:39:45 AM UTC-8 harald@gmx.net 
wrote:

> The containerd project  doesn't 
> publish a Python package for its containerd gRPC-based API, so I want to 
> generate Python modules for it myself. For API version 1.3 of containerd, 
> the .proto files can be found in 
> https://github.com/containerd/containerd/tree/release/1.3/api. 
> Unfortunately, I'm hitting two road blocks when trying to generate Python 
> modules from the API .proto files and when to run them.
>
> For example, the some of the containerd API .proto files reference 
> protobuf plugin .proto files using an (for lack of a better term on my 
> side) absolute import path: api/events/container.proto 
> 
>  does 
> an:
>
> import weak "
> github.com/containerd/containerd/protobuf/plugin/fieldpath.proto";
>
> However, the Python grpc compiler always wants to resolve such references 
> in the local file system and containerd's source have 
> ./protobuf/plugin/fieldpath.proto 
> 
>  -- 
> so this won't ever resolve correctly (using -I ...), because it lacks the 
> github.lcom/containerd/containerd path elements. Please note that the 
> Python grpc protocol compiler resolves such import paths in form of 
> chopping down the full import path into parts and then searching its -I 
> ... include directories.
>
> Trying the lazy route by simply copying over these sources into 
> inside vendor/github.com/... inside the containerd source tree will later 
> cause *runtime errors *when trying to use the generated Python modules: 
> this is because the grpc compiler considers the same containerd API .proto 
> file in two locations (paths) to be separate instances. In consequence we 
> get duplicate modules which unfortunately now try to register with grpc for 
> the same protocol element names. Consequently, the gRPC Python runtime thus 
> throws an error and terminates.
>
> How can I correctly get this resolved when using python3 -m 
> grpc.tools.protoc ...? What am I missing here or getting wrong?
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/ce257795-7a29-4bdb-8bc9-57538c00eb33n%40googlegroups.com.