Re: [protobuf] Generating Code With Parent Directory Imports?

['Feng Xiao' via Protocol Buffers]

On Wed, Jan 31, 2018 at 2:57 PM, Jonnny  wrote:

> By 'Protobuf Python Library' you mean the Python Package called `protobuf
> ? I think It's already included in my package as dependency. But Google
> protobuf is just one example. Most people around here using Scala, and they
> all have 'scalapb' imported into their proto files. The path is `*scalapb*
> /*scalapb*.*proto`. *It's not something I can chance as it's a third
> party proto, and it's not a Google protobuf so using the `protobuf` package
> won't solve this issue. I can manually add it into my `/usr/local/include`
> folder, but this mean the package is not self-contained and the user needs
> to compile his own scalapb pb2 file, and manually add it to
> `/usr/local/include` if he wants my package to work.
>
Is there not a Pypi package containing scalapb/scalapb_pb2.py already? How
about ask the author of scalapb/scalapb.proto to provide such a python
package?


>
> It means again, I have to keep the `scalapb` folder set into the root of
> my project, or play around with PYTHONPATH. I wasn't able to find online
> Python projects/packages that uses Python complied proto code, so I was
> thinking perhaps there is a way around it. But if there is not a better
> way, and that's consider the 'standard', I'm guessing I can do one of the
> three below:
>
> 1. On a normal project, `scalapb` should be sitting on the root of the
> project (ie, the folder where I'm running the script from).
> 2. Packages are more complex, because the 'root' includes things like
> `setup.py` and `README.srt`, but not the actual package code. The package
> sits in a directory inside the root Folder. For example in the Request
> package (https://github.com/requests/requests),  the actual code run from
> within the `request` folder. So even if I'll keep the generated code
> directory structure in the `requests` folder, it won't work necause the
> relative path starts from the Root folder and not `requests` folder. Adding
> the folder structure into the root of the project is messy (because you
> need to defines those as package in `setup.py`, and perhaps even supply
> __init__.py file).
> 3. Playing with PYTHONPATH like so:
>
>
> import sys
> sys.path.append('./my_package_name/proto')
>
>
> This code need to be called every-time the package is called (probably need 
> to sit inside the `__init__.py I'm guessing).
>
>
> There is a 4th option, but one that I do not like much. In case I have
> small amount of `pb2` files, I can manually edit the `pb2` imports by
> adding my folder after the code is compiled.
>
> I would just go ahead with (1). What's so bad about having a scalapb
directory in your root directory? If you are concerned with conflicts with
other project that may also include scalapb, you can just create a separate
python package for the scalapb/scalapb_pb2.py, deploy it to pypi, and share
it with all other projects that depend on the same thing.


>
>
>
>
>
> On Wednesday, January 31, 2018 at 11:39:08 PM UTC+2, Feng Xiao wrote:
>>
>>
>>
>> On Wed, Jan 31, 2018 at 8:54 AM, Jonnny  wrote:
>>
>>> Hello everyone,
>>>
>>> I have a couple of micro-services, and each has couple of proto files.
>>> It's common for one micro-service proto file to reference another service
>>> proto file using import like so:
>>>
>>> import "google/protobuf/struct.proto";
>>> import "myService/v1/message.proto";
>>> import "myOtherService/v2/parser.proto";
>>>
>>> We use a Gradle script that's responsible on creating the folder
>>> structure and fetch all proto dependencies from different git repositories.
>>> So in the example above the Gradle script will output will be a folder
>>> named `google/protobuff`, `myServicev1/v1` and `myOtherService/v2` - each
>>> of the folders will hold the proto files needed. Once the folder structure
>>> is OK, I can use `protoc` to generate the Python code.
>>>
>>> Here's where the problem begins. The generated `_pb2` code import
>>> modules based on the folder structure. So the import inside the `_pb2` file
>>> will look like so:
>>>
>>> from google.protobuf import struct_pb2 as 
>>> google_dot_protobuf_dot_struct__pb2
>>>
>>>
>>> That's a relative path import, and it means that in order for my Python
>>> program to work, the root folder where the Python script running from,
>>> should have the `google/protobuff` folder for this import to work.Now
>>> because I have multiple micro-services, I don't appreciated the fact I have
>>> to add 3 new folders in my main Python project (it's even more painful when
>>> you create a Pip package). What I was hoping to do is to open a directory
>>> called `proto`, and add all that directory structure into it - but because
>>> imports are relative, this won't work. I tried to to play with couple of
>>> `protoc` options like `PROTO_PATH`, or run `protoc` from a parent
>>> directory, so the `_pb2` files will get created with a different relative
>>> path I can work with like so:
>>>
>>> from 

Re: [protobuf] Generating Code With Parent Directory Imports?

[Jonnny]

By 'Protobuf Python Library' you mean the Python Package called `protobuf ? 
I think It's already included in my package as dependency. But Google 
protobuf is just one example. Most people around here using Scala, and they 
all have 'scalapb' imported into their proto files. The path is `*scalapb*/
*scalapb*.*proto`. *It's not something I can chance as it's a third party 
proto, and it's not a Google protobuf so using the `protobuf` package won't 
solve this issue. I can manually add it into my `/usr/local/include` 
folder, but this mean the package is not self-contained and the user needs 
to compile his own scalapb pb2 file, and manually add it to 
`/usr/local/include` if he wants my package to work. 

It means again, I have to keep the `scalapb` folder set into the root of my 
project, or play around with PYTHONPATH. I wasn't able to find online 
Python projects/packages that uses Python complied proto code, so I was 
thinking perhaps there is a way around it. But if there is not a better 
way, and that's consider the 'standard', I'm guessing I can do one of the 
three below:

1. On a normal project, `scalapb` should be sitting on the root of the 
project (ie, the folder where I'm running the script from).
2. Packages are more complex, because the 'root' includes things like 
`setup.py` and `README.srt`, but not the actual package code. The package 
sits in a directory inside the root Folder. For example in the Request 
package (https://github.com/requests/requests),  the actual code run from 
within the `request` folder. So even if I'll keep the generated code 
directory structure in the `requests` folder, it won't work necause the 
relative path starts from the Root folder and not `requests` folder. Adding 
the folder structure into the root of the project is messy (because you 
need to defines those as package in `setup.py`, and perhaps even supply 
__init__.py file).
3. Playing with PYTHONPATH like so:


import sys
sys.path.append('./my_package_name/proto')


This code need to be called every-time the package is called (probably need to 
sit inside the `__init__.py I'm guessing).


There is a 4th option, but one that I do not like much. In case I have 
small amount of `pb2` files, I can manually edit the `pb2` imports by 
adding my folder after the code is compiled.






On Wednesday, January 31, 2018 at 11:39:08 PM UTC+2, Feng Xiao wrote:
>
>
>
> On Wed, Jan 31, 2018 at 8:54 AM, Jonnny  
> wrote:
>
>> Hello everyone,
>>
>> I have a couple of micro-services, and each has couple of proto files. 
>> It's common for one micro-service proto file to reference another service 
>> proto file using import like so:
>>
>> import "google/protobuf/struct.proto";
>> import "myService/v1/message.proto";
>> import "myOtherService/v2/parser.proto";
>>
>> We use a Gradle script that's responsible on creating the folder 
>> structure and fetch all proto dependencies from different git repositories. 
>> So in the example above the Gradle script will output will be a folder 
>> named `google/protobuff`, `myServicev1/v1` and `myOtherService/v2` - each 
>> of the folders will hold the proto files needed. Once the folder structure 
>> is OK, I can use `protoc` to generate the Python code.
>>
>> Here's where the problem begins. The generated `_pb2` code import modules 
>> based on the folder structure. So the import inside the `_pb2` file will 
>> look like so:
>>
>> from google.protobuf import struct_pb2 as 
>> google_dot_protobuf_dot_struct__pb2 
>>
>> That's a relative path import, and it means that in order for my Python 
>> program to work, the root folder where the Python script running from, 
>> should have the `google/protobuff` folder for this import to work.Now 
>> because I have multiple micro-services, I don't appreciated the fact I have 
>> to add 3 new folders in my main Python project (it's even more painful when 
>> you create a Pip package). What I was hoping to do is to open a directory 
>> called `proto`, and add all that directory structure into it - but because 
>> imports are relative, this won't work. I tried to to play with couple of 
>> `protoc` options like `PROTO_PATH`, or run `protoc` from a parent 
>> directory, so the `_pb2` files will get created with a different relative 
>> path I can work with like so:
>>
>> from proto.google.protobuf import struct_pb2 as 
>> google_dot_protobuf_dot_struct__pb2 
>>
>> But I wasn't able to do so. A dirty option would be adding the `proto` 
>> folder into my `PYTHONPATH`. It's not pretty, and I'm guessing there is 
>> probably a simple solution for that I'm missing with `protoc`.
>>
> You can probably change the path for your own message.proto/parser.proto, 
> but for protobuf's google/protobuf/struct.proto (and any other protos under 
> google/protobuf), it's the only option to import it as 
> "google/protobuf/struct.proto". Nothing else would work. It just has to be 
> imported as "google/protobuf/struct.proto" in any .proto file and 

Re: [protobuf] Generating Code With Parent Directory Imports?

['Feng Xiao' via Protocol Buffers]

On Wed, Jan 31, 2018 at 8:54 AM, Jonnny  wrote:

> Hello everyone,
>
> I have a couple of micro-services, and each has couple of proto files.
> It's common for one micro-service proto file to reference another service
> proto file using import like so:
>
> import "google/protobuf/struct.proto";
> import "myService/v1/message.proto";
> import "myOtherService/v2/parser.proto";
>
> We use a Gradle script that's responsible on creating the folder structure
> and fetch all proto dependencies from different git repositories. So in the
> example above the Gradle script will output will be a folder named
> `google/protobuff`, `myServicev1/v1` and `myOtherService/v2` - each of the
> folders will hold the proto files needed. Once the folder structure is OK,
> I can use `protoc` to generate the Python code.
>
> Here's where the problem begins. The generated `_pb2` code import modules
> based on the folder structure. So the import inside the `_pb2` file will
> look like so:
>
> from google.protobuf import struct_pb2 as google_dot_protobuf_dot_struct__pb2
>
>
> That's a relative path import, and it means that in order for my Python
> program to work, the root folder where the Python script running from,
> should have the `google/protobuff` folder for this import to work.Now
> because I have multiple micro-services, I don't appreciated the fact I have
> to add 3 new folders in my main Python project (it's even more painful when
> you create a Pip package). What I was hoping to do is to open a directory
> called `proto`, and add all that directory structure into it - but because
> imports are relative, this won't work. I tried to to play with couple of
> `protoc` options like `PROTO_PATH`, or run `protoc` from a parent
> directory, so the `_pb2` files will get created with a different relative
> path I can work with like so:
>
> from proto.google.protobuf import struct_pb2 as 
> google_dot_protobuf_dot_struct__pb2
>
>
> But I wasn't able to do so. A dirty option would be adding the `proto`
> folder into my `PYTHONPATH`. It's not pretty, and I'm guessing there is
> probably a simple solution for that I'm missing with `protoc`.
>
You can probably change the path for your own message.proto/parser.proto,
but for protobuf's google/protobuf/struct.proto (and any other protos under
google/protobuf), it's the only option to import it as
"google/protobuf/struct.proto". Nothing else would work. It just has to be
imported as "google/protobuf/struct.proto" in any .proto file and has to be
imported as "google.protobuf.struct_pb2" in any python file. It's part of
protobuf public API and won't (and can't) be changed.

Where is protobuf python library installed on your machine? If you have it
installed, google.protobuf.struct_pb2 should be part of it and you
shouldn't need to put google/protobuf/struct_pb2.py in your own project
directory.


>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Generating Code With Parent Directory Imports?

[Jonnny]

Hello everyone,

I have a couple of micro-services, and each has couple of proto files. It's 
common for one micro-service proto file to reference another service proto 
file using import like so:

import "google/protobuf/struct.proto";
import "myService/v1/message.proto";
import "myOtherService/v2/parser.proto";

We use a Gradle script that's responsible on creating the folder structure 
and fetch all proto dependencies from different git repositories. So in the 
example above the Gradle script will output will be a folder named 
`google/protobuff`, `myServicev1/v1` and `myOtherService/v2` - each of the 
folders will hold the proto files needed. Once the folder structure is OK, 
I can use `protoc` to generate the Python code.

Here's where the problem begins. The generated `_pb2` code import modules 
based on the folder structure. So the import inside the `_pb2` file will 
look like so:

from google.protobuf import struct_pb2 as 
google_dot_protobuf_dot_struct__pb2 

That's a relative path import, and it means that in order for my Python 
program to work, the root folder where the Python script running from, 
should have the `google/protobuff` folder for this import to work.Now 
because I have multiple micro-services, I don't appreciated the fact I have 
to add 3 new folders in my main Python project (it's even more painful when 
you create a Pip package). What I was hoping to do is to open a directory 
called `proto`, and add all that directory structure into it - but because 
imports are relative, this won't work. I tried to to play with couple of 
`protoc` options like `PROTO_PATH`, or run `protoc` from a parent 
directory, so the `_pb2` files will get created with a different relative 
path I can work with like so:

from proto.google.protobuf import struct_pb2 as 
google_dot_protobuf_dot_struct__pb2 

But I wasn't able to do so. A dirty option would be adding the `proto` 
folder into my `PYTHONPATH`. It's not pretty, and I'm guessing there is 
probably a simple solution for that I'm missing with `protoc`.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.