Re: [DISCUSS] Separate the Python Functions from the installation of the Python client

[Yunze Xu]

Hi Pengcheng,

> I think there is no need to setup a "functions" lib?

Yes. Even before 3.2.0 (not released yet), users can still set up the
functions by installing `pulsar-client` without the `functions` extra
component and installing the other dependencies manually.

Thanks,
Yunze

On Thu, Apr 20, 2023 at 10:49 AM Pengcheng Jiang
 wrote:
>
> Hi Yunze,
>
> +1 for separating
>
> And for the migration progress:
> 1. since the `pulsar-functions` lib doesn't have any code but just a list
> of dependencies, I think there is no need to setup a "functions" lib?
> 2. we should update docs about what dependencies are required for Python
> functions
> 3. the Dockerfile of Pulsar image should be updated to install the required
> python dependencies so it can start python function in process runtime mode
> 4. the python runner image's Dockerfile should be updated to install the
> required python dependencies so it can run in k8s runtime mode
>
> Regards,
> Pengcheng Jiang
>
>
> Yunze Xu  于2023年4月20日周四 10:46写道：
>
> > Hi Neng,
> >
> > I think currently a simple solution is to document which dependencies
> > should users install to use the Python Functions.
> >
> > Before:
> >
> > ```
> > # For version < 3.2.0
> > pip install pulsar-client[functions]
> > ```
> >
> > Now:
> >
> > ```
> > # For version >= 3.2.0
> > pip install pulsar-client
> > pip install grpc-io
> > # Other dependencies...
> > ```
> >
> > In future, we can provide a separate PyPI package like `pulsar-functions`.
> >
> > BTW, currently the functions extra dependencies cannot be installed
> > for Python 3.10 and 3.11. I have tested the following images with the
> > `pip install 'pulsar-client[functions]==3.1.0'` command.
> > - python:3.11.2-bullseye
> > - python:3.10.11-bullseye
> >
> > The reason is the version incompatibility of grpcio.
> >
> > > ERROR: Failed building wheel for grpcio
> >
> > Though I only tested 3.1.0, since the dependencies never changed, they
> > should also fail for older Python client releases like 2.10.0.
> >
> > Thanks,
> > Yunze
> >
> > On Thu, Apr 20, 2023 at 12:26 AM Neng Lu  wrote:
> > >
> > > Hi Yunze,
> > >
> > > +1 for separating Python client and Python Pulsar Functions pip
> > installation.
> > > On the Java side, the client lib and functions lib are also published
> > separately.
> > >
> > > My concern is how the migration progress should look like,
> > > 1. we need to set up functions lib so that users can install it using
> > `pip install pulsar-functions`
> > > 2. the current `pip install pulsar-client[functions]` should prompt user
> > to use the new way
> > > 3. all docs need to be updated
> > > 4. for historical versions, what can we do?
> > >
> > >
> > >
> > > On 2023/04/19 15:23:49 Yunze Xu wrote:
> > > > Hi all,
> > > >
> > > > The Python client has been separated since PIP-209 [1] and now the
> > > > Python client is maintained in a separated repository [2]. However,
> > > > the Python Function is still maintained in the main repo [3].
> > > >
> > > > Currently, we can install the Python client with the following ways:
> > > > 1. pip install pulsar-client
> > > > 2. pip install pulsar-client[avro]
> > > > 3. pip install pulsar-client[functions]
> > > > 4. pip install pulsar-client[all]
> > > >
> > > > See [4] for the difference. However, for the 3rd and 4th ways, it
> > > > installs all the dependencies required by the Python Functions.
> > > > However, they are broken for the recent releases because of the
> > > > outdated dependencies [5]. However, these dependencies are from the
> > > > Python Functions [3], not the Python client library itself. Also,
> > > > there are no tests in the Python client repo [2] for these functions
> > > > so these dependencies cannot be verified.
> > > >
> > > > IMO, these dependencies should be maintained in the directory of the
> > > > Python Functions. We should not rely on the Python client to install
> > > > the dependencies for the Python Functions.
> > > >
> > > > Therefore, my suggestion is to drop the 3rd and 4th installation
> > > > methods in future releases of the Python client. After that, we should
> > > > update the scripts in the main repo to install the Python Functions in
> > > > [3].
> > > >
> > > > I'm not familiar with the Pulsar Functions, so feel free to show your
> > > > suggestions if any of you have any concerns.
> > > >
> > > > [1] https://github.com/apache/pulsar/pull/17881
> > > > [2] http://github.com/apache/pulsar-client-python
> > > > [3]
> > https://github.com/apache/pulsar/tree/master/pulsar-functions/instance/src/main/python
> > > > [4] https://pulsar.apache.org/docs/2.11.x/client-libraries-python/
> > > > [5]
> > https://github.com/apache/pulsar-client-python/blob/a6476d9c45508f55a7af4b25001038a8e3a27489/setup.py#L80-L88
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> >
>
>
> --
>
> 
>
> Pengcheng Jiang
>
> Software Engineer
>
> e: pengcheng.ji...@streamnative.io
>
> p: 13540631948
>
> streamnative.io
>
> 

Re: [DISCUSS] Separate the Python Functions from the installation of the Python client

[Pengcheng Jiang]

Hi Yunze,

+1 for separating

And for the migration progress:
1. since the `pulsar-functions` lib doesn't have any code but just a list
of dependencies, I think there is no need to setup a "functions" lib?
2. we should update docs about what dependencies are required for Python
functions
3. the Dockerfile of Pulsar image should be updated to install the required
python dependencies so it can start python function in process runtime mode
4. the python runner image's Dockerfile should be updated to install the
required python dependencies so it can run in k8s runtime mode

Regards,
Pengcheng Jiang


Yunze Xu  于2023年4月20日周四 10:46写道：

> Hi Neng,
>
> I think currently a simple solution is to document which dependencies
> should users install to use the Python Functions.
>
> Before:
>
> ```
> # For version < 3.2.0
> pip install pulsar-client[functions]
> ```
>
> Now:
>
> ```
> # For version >= 3.2.0
> pip install pulsar-client
> pip install grpc-io
> # Other dependencies...
> ```
>
> In future, we can provide a separate PyPI package like `pulsar-functions`.
>
> BTW, currently the functions extra dependencies cannot be installed
> for Python 3.10 and 3.11. I have tested the following images with the
> `pip install 'pulsar-client[functions]==3.1.0'` command.
> - python:3.11.2-bullseye
> - python:3.10.11-bullseye
>
> The reason is the version incompatibility of grpcio.
>
> > ERROR: Failed building wheel for grpcio
>
> Though I only tested 3.1.0, since the dependencies never changed, they
> should also fail for older Python client releases like 2.10.0.
>
> Thanks,
> Yunze
>
> On Thu, Apr 20, 2023 at 12:26 AM Neng Lu  wrote:
> >
> > Hi Yunze,
> >
> > +1 for separating Python client and Python Pulsar Functions pip
> installation.
> > On the Java side, the client lib and functions lib are also published
> separately.
> >
> > My concern is how the migration progress should look like,
> > 1. we need to set up functions lib so that users can install it using
> `pip install pulsar-functions`
> > 2. the current `pip install pulsar-client[functions]` should prompt user
> to use the new way
> > 3. all docs need to be updated
> > 4. for historical versions, what can we do?
> >
> >
> >
> > On 2023/04/19 15:23:49 Yunze Xu wrote:
> > > Hi all,
> > >
> > > The Python client has been separated since PIP-209 [1] and now the
> > > Python client is maintained in a separated repository [2]. However,
> > > the Python Function is still maintained in the main repo [3].
> > >
> > > Currently, we can install the Python client with the following ways:
> > > 1. pip install pulsar-client
> > > 2. pip install pulsar-client[avro]
> > > 3. pip install pulsar-client[functions]
> > > 4. pip install pulsar-client[all]
> > >
> > > See [4] for the difference. However, for the 3rd and 4th ways, it
> > > installs all the dependencies required by the Python Functions.
> > > However, they are broken for the recent releases because of the
> > > outdated dependencies [5]. However, these dependencies are from the
> > > Python Functions [3], not the Python client library itself. Also,
> > > there are no tests in the Python client repo [2] for these functions
> > > so these dependencies cannot be verified.
> > >
> > > IMO, these dependencies should be maintained in the directory of the
> > > Python Functions. We should not rely on the Python client to install
> > > the dependencies for the Python Functions.
> > >
> > > Therefore, my suggestion is to drop the 3rd and 4th installation
> > > methods in future releases of the Python client. After that, we should
> > > update the scripts in the main repo to install the Python Functions in
> > > [3].
> > >
> > > I'm not familiar with the Pulsar Functions, so feel free to show your
> > > suggestions if any of you have any concerns.
> > >
> > > [1] https://github.com/apache/pulsar/pull/17881
> > > [2] http://github.com/apache/pulsar-client-python
> > > [3]
> https://github.com/apache/pulsar/tree/master/pulsar-functions/instance/src/main/python
> > > [4] https://pulsar.apache.org/docs/2.11.x/client-libraries-python/
> > > [5]
> https://github.com/apache/pulsar-client-python/blob/a6476d9c45508f55a7af4b25001038a8e3a27489/setup.py#L80-L88
> > >
> > > Thanks,
> > > Yunze
> > >
>


-- 



Pengcheng Jiang

Software Engineer

e: pengcheng.ji...@streamnative.io

p: 13540631948

streamnative.io

Re: [DISCUSS] Separate the Python Functions from the installation of the Python client

[Yunze Xu]

Hi Neng,

I think currently a simple solution is to document which dependencies
should users install to use the Python Functions.

Before:

```
# For version < 3.2.0
pip install pulsar-client[functions]
```

Now:

```
# For version >= 3.2.0
pip install pulsar-client
pip install grpc-io
# Other dependencies...
```

In future, we can provide a separate PyPI package like `pulsar-functions`.

BTW, currently the functions extra dependencies cannot be installed
for Python 3.10 and 3.11. I have tested the following images with the
`pip install 'pulsar-client[functions]==3.1.0'` command.
- python:3.11.2-bullseye
- python:3.10.11-bullseye

The reason is the version incompatibility of grpcio.

> ERROR: Failed building wheel for grpcio

Though I only tested 3.1.0, since the dependencies never changed, they
should also fail for older Python client releases like 2.10.0.

Thanks,
Yunze

On Thu, Apr 20, 2023 at 12:26 AM Neng Lu  wrote:
>
> Hi Yunze,
>
> +1 for separating Python client and Python Pulsar Functions pip installation.
> On the Java side, the client lib and functions lib are also published 
> separately.
>
> My concern is how the migration progress should look like,
> 1. we need to set up functions lib so that users can install it using `pip 
> install pulsar-functions`
> 2. the current `pip install pulsar-client[functions]` should prompt user to 
> use the new way
> 3. all docs need to be updated
> 4. for historical versions, what can we do?
>
>
>
> On 2023/04/19 15:23:49 Yunze Xu wrote:
> > Hi all,
> >
> > The Python client has been separated since PIP-209 [1] and now the
> > Python client is maintained in a separated repository [2]. However,
> > the Python Function is still maintained in the main repo [3].
> >
> > Currently, we can install the Python client with the following ways:
> > 1. pip install pulsar-client
> > 2. pip install pulsar-client[avro]
> > 3. pip install pulsar-client[functions]
> > 4. pip install pulsar-client[all]
> >
> > See [4] for the difference. However, for the 3rd and 4th ways, it
> > installs all the dependencies required by the Python Functions.
> > However, they are broken for the recent releases because of the
> > outdated dependencies [5]. However, these dependencies are from the
> > Python Functions [3], not the Python client library itself. Also,
> > there are no tests in the Python client repo [2] for these functions
> > so these dependencies cannot be verified.
> >
> > IMO, these dependencies should be maintained in the directory of the
> > Python Functions. We should not rely on the Python client to install
> > the dependencies for the Python Functions.
> >
> > Therefore, my suggestion is to drop the 3rd and 4th installation
> > methods in future releases of the Python client. After that, we should
> > update the scripts in the main repo to install the Python Functions in
> > [3].
> >
> > I'm not familiar with the Pulsar Functions, so feel free to show your
> > suggestions if any of you have any concerns.
> >
> > [1] https://github.com/apache/pulsar/pull/17881
> > [2] http://github.com/apache/pulsar-client-python
> > [3] 
> > https://github.com/apache/pulsar/tree/master/pulsar-functions/instance/src/main/python
> > [4] https://pulsar.apache.org/docs/2.11.x/client-libraries-python/
> > [5] 
> > https://github.com/apache/pulsar-client-python/blob/a6476d9c45508f55a7af4b25001038a8e3a27489/setup.py#L80-L88
> >
> > Thanks,
> > Yunze
> >

Re: [DISCUSS] Separate the Python Functions from the installation of the Python client

[Neng Lu]

Hi Yunze,

+1 for separating Python client and Python Pulsar Functions pip installation.
On the Java side, the client lib and functions lib are also published 
separately.

My concern is how the migration progress should look like,
1. we need to set up functions lib so that users can install it using `pip 
install pulsar-functions`
2. the current `pip install pulsar-client[functions]` should prompt user to use 
the new way
3. all docs need to be updated
4. for historical versions, what can we do?



On 2023/04/19 15:23:49 Yunze Xu wrote:
> Hi all,
> 
> The Python client has been separated since PIP-209 [1] and now the
> Python client is maintained in a separated repository [2]. However,
> the Python Function is still maintained in the main repo [3].
> 
> Currently, we can install the Python client with the following ways:
> 1. pip install pulsar-client
> 2. pip install pulsar-client[avro]
> 3. pip install pulsar-client[functions]
> 4. pip install pulsar-client[all]
> 
> See [4] for the difference. However, for the 3rd and 4th ways, it
> installs all the dependencies required by the Python Functions.
> However, they are broken for the recent releases because of the
> outdated dependencies [5]. However, these dependencies are from the
> Python Functions [3], not the Python client library itself. Also,
> there are no tests in the Python client repo [2] for these functions
> so these dependencies cannot be verified.
> 
> IMO, these dependencies should be maintained in the directory of the
> Python Functions. We should not rely on the Python client to install
> the dependencies for the Python Functions.
> 
> Therefore, my suggestion is to drop the 3rd and 4th installation
> methods in future releases of the Python client. After that, we should
> update the scripts in the main repo to install the Python Functions in
> [3].
> 
> I'm not familiar with the Pulsar Functions, so feel free to show your
> suggestions if any of you have any concerns.
> 
> [1] https://github.com/apache/pulsar/pull/17881
> [2] http://github.com/apache/pulsar-client-python
> [3] 
> https://github.com/apache/pulsar/tree/master/pulsar-functions/instance/src/main/python
> [4] https://pulsar.apache.org/docs/2.11.x/client-libraries-python/
> [5] 
> https://github.com/apache/pulsar-client-python/blob/a6476d9c45508f55a7af4b25001038a8e3a27489/setup.py#L80-L88
> 
> Thanks,
> Yunze
> 

[DISCUSS] Separate the Python Functions from the installation of the Python client

[Yunze Xu]

Hi all,

The Python client has been separated since PIP-209 [1] and now the
Python client is maintained in a separated repository [2]. However,
the Python Function is still maintained in the main repo [3].

Currently, we can install the Python client with the following ways:
1. pip install pulsar-client
2. pip install pulsar-client[avro]
3. pip install pulsar-client[functions]
4. pip install pulsar-client[all]

See [4] for the difference. However, for the 3rd and 4th ways, it
installs all the dependencies required by the Python Functions.
However, they are broken for the recent releases because of the
outdated dependencies [5]. However, these dependencies are from the
Python Functions [3], not the Python client library itself. Also,
there are no tests in the Python client repo [2] for these functions
so these dependencies cannot be verified.

IMO, these dependencies should be maintained in the directory of the
Python Functions. We should not rely on the Python client to install
the dependencies for the Python Functions.

Therefore, my suggestion is to drop the 3rd and 4th installation
methods in future releases of the Python client. After that, we should
update the scripts in the main repo to install the Python Functions in
[3].

I'm not familiar with the Pulsar Functions, so feel free to show your
suggestions if any of you have any concerns.

[1] https://github.com/apache/pulsar/pull/17881
[2] http://github.com/apache/pulsar-client-python
[3] 
https://github.com/apache/pulsar/tree/master/pulsar-functions/instance/src/main/python
[4] https://pulsar.apache.org/docs/2.11.x/client-libraries-python/
[5] 
https://github.com/apache/pulsar-client-python/blob/a6476d9c45508f55a7af4b25001038a8e3a27489/setup.py#L80-L88

Thanks,
Yunze