Response inline
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Fri, Mar 27, 2026 at 1:01 PM Nicholas Chammas <[email protected]> wrote: > > On Mar 27, 2026, at 12:31 PM, Holden Karau <[email protected]> wrote: > > One possibility would be to make the pinned version optional (eg > pyspark[pinned]) or publish a separate constraints file for people to > optionally use with -c? > > > Perhaps I am misunderstanding your proposal, Holden, but this is possible > today for people using modern Python packaging workflows that use lock > files. In fact, it happens automatically; all transitive dependencies are > pinned in the lock file, and this is by design. > So for someone installing a fresh venv with uv/pip/or conda where does this come from? The idea here is we provide the versions we used during the release stage so if folks want a “known safe” initial starting point for a new env they’ve got one. > > Furthermore, it is straightforward to add additional restrictions to your > project spec (i.e. pyproject.toml) so that when the packaging tool builds > the lock file, it does it with whatever restrictions you want that are > specific to your project. That could include specific versions or version > ranges of libraries to exclude, for example. > Yes, but as it stands we leave it to the end user to start from scratch picking these versions, we can make their lives simpler by providing the versions we tested against with a lock file they can choose to use, ignore, or update to their desired versions and include. Also for interactive workloads I more often see a bare requirements file or even pip installs in nb cells (but this could be sample bias). > > I had to do this, for example, on a personal project that used PySpark > Connect but which was pulling in a version of grpc that was generating a > lot of log noise > <https://github.com/grpc/grpc/issues/38336#issuecomment-2588422915>. I > pinned the version of grpc in my project file and let the packaging tool > resolve all the requirements across PySpark Connect and my custom > restrictions. > > Nick > >
