[Numpy-discussion] ANN: Python-Blosc2 3.0.0 release candidate 1 is out!
Announcing Python-Blosc2 3.0.0-rc1
==
The Blosc development team is pleased to announce the first release release of
Python-Blosc2 3.0.0. In this release, we have focused on the making of a
compute engine that can work with compressed data in a NumPy-like fashion.
You can think of Python-Blosc2 3.0 as a replacement of numexpr, but better :-)
As always, we would like to get feedback from the community before the final
release. We are providing binary wheels that you can easily install from PyPI
with:
pip install blosc2==3.0.0rc1
For more info, you can have a look at the release notes in:
https://github.com/Blosc/python-blosc2/releases
Docs and examples are available in the documentation site:
https://www.blosc.org/python-blosc2
Code example::
import blosc2
from time import time
# Create some data operands
N = 10_000
a = blosc2.linspace(0, 1, N * N, dtype="float32", shape=(N, N))
b = blosc2.linspace(1, 2, N * N, shape=(N, N))
c = blosc2.linspace(-10, 10, N * N, shape=(N, N))
# Expression
t0 = time()
expr = ((a**3 + blosc2.sin(c * 2)) < b) & (c > 0)
print(f"Time to create expression: {time()-t0:.5f}")
# Evaluate and get a NDArray as result
t0 = time()
out = expr.compute()
print(f"Time to compute: {time()-t0:.5f}")
This will output something like::
Time to create expression: 0.00041
Time to compute: 0.56215
Note that the expression is computed lazily, only when the `compute`
method is called.
BTW, I'll be teaching about Python-Blosc2 3.0 in my forthcoming
tutorial at PyData Global 2024
(https://pydata.org/global2024/schedule). Be sure to be there for an
exciting introduction to the bells and whistles of the shiny new
computation engine and its enhanced capabilities to deal with large
datasets.
Thanks and see you there!
--
Francesc Alted
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] ANN: Python-Blosc2 3.0, with a new compute engine, is out
Announcing Python-Blosc2 3.0.0 (final release)
==
The Blosc development team is pleased to announce the final release of
Python-Blosc2 3.0.0. Now, we will be producing conda(-forge) packages,
as well as providing wheels for the most common platforms, as usual.
In this major release, we are adding a new compute engine tuned for compressed
data that can be stored either in-memory, on-disk or on the network (via the
Caterva2 library (https://github.com/ironArray/Caterva2/).
You can think of Python-Blosc2 3.0 as an extension of NumPy/numexpr that:
- Can deal with ndarrays (optionally) compressed using first-class
codecs & filters.
- Performs many kind of math expressions, including reductions,
indexing and more.
- Supports broadcasting operations.
- Supports NumPy ufunc mechanism, allowing to mix and match NumPy and
Blosc2 computations.
- Integrates with Numba and Cython via UDFs (User Defined Functions).
- Adheres to modern NumPy casting rules way better than numexpr.
- Computes expressions lazily, only when needed, and can be stored for
later use.
Install it with::
pip install blosc2==3.0.0 # if you prefer wheels
conda install -c conda-forge python-blosc2 mkl # if you prefer
conda and MKL
For more info, you can have a look at the release notes in:
https://github.com/Blosc/python-blosc2/releases
Code example::
from time import time
import blosc2
import numpy as np
# Create some data operands
N = 20_000
a = blosc2.linspace(0, 1, N * N, dtype="float32", shape=(N, N))
b = blosc2.linspace(1, 2, N * N, shape=(N, N))
c = blosc2.linspace(-10, 10, N) # broadcasting is supported
# Expression
t0 = time()
expr = ((a**3 + blosc2.sin(c * 2)) < b) & (c > 0)
print(f"Time to create expression: {time()-t0:.5f}")
# Evaluate while reducing (yep, reductions are in) along axis 1
t0 = time()
out = blosc2.sum(expr, axis=1)
t1 = time() - t0
print(f"Time to compute with Blosc2: {t1:.5f}")
# Evaluate using NumPy
na, nb, nc = a[:], b[:], c[:]
t0 = time()
nout = np.sum(((na**3 + np.sin(nc * 2)) < nb) & (nc > 0), axis=1)
t2 = time() - t0
print(f"Time to compute with NumPy: {t2:.5f}")
print(f"Speedup: {t2/t1:.2f}x")
assert np.all(out == nout)
print("All results are equal!")
This will output something like (using an Intel i9-13900X CPU here)::
Time to create expression: 0.00033
Time to compute with Blosc2: 0.46387
Time to compute with NumPy: 2.57469
Speedup: 5.55x
All results are equal!
See a more in-depth example, explaining why Python-Blosc2 is so fast, at:
https://www.blosc.org/python-blosc2/getting_started/overview.html#operating-with-ndarrays
Docs
Read some of our tutorials on how to perform advanced computations at:
https://www.blosc.org/python-blosc2/getting_started/tutorials
See also materials for our recent PyData Global 2024 tutorial at:
https://github.com/Blosc/Python-Blosc2-3.0-tutorial
And the full documentation at:
https://www.blosc.org/python-blosc2
Enjoy!
- Blosc Development Team
Compress Better, Compute Bigger
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] ANN: Python-Blosc2 3.3.4 released
Hi, This is a bugfix release, with some minor optimizations. We further improved the correct chaining of *string* lazy expressions (to allow operands with more diverse data types). In addition, both indexing and where expressions are now supported within *string* lazy expressions. Finally, casting rules have been improved to be more consistent with NumPy. You can think of Python-Blosc2 3.x as an extension of NumPy/numexpr that: - Can deal with ndarrays compressed using first-class codecs & filters. - Performs many kind of math expressions, including reductions, indexing... - Supports broadcasting operations. - Supports NumPy ufunc mechanism: mix and match NumPy and Blosc2 computations. - Integrates with Numba and Cython via UDFs (User Defined Functions). - Adheres to modern NumPy casting rules way better than numexpr. - Supports linear algebra operations (like ``blosc2.matmul()``). Install it with:: pip install blosc2 --update # if you prefer wheels conda install -c conda-forge python-blosc2 mkl # if you prefer conda and MKL For more info, you can have a look at the release notes in: https://github.com/Blosc/python-blosc2/releases Have fun! -- Francesc Alted ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: arch...@mail-archive.com
[Numpy-discussion] ANN: NumExpr 2.11.0, with free-threaded support, is out!
= Announcing NumExpr 2.11.0 = Hi everyone, NumExpr 2.11.0 Initial support for free-threaded Python 3.13t has been added. This is still experimental, so please report any issues you find. Finally, Python 3.10 is now the minimum supported version. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.10.2 to 2.11.0 - * Initial support for free-threaded Python 3.13t has been added. This is still experimental, so please report any issues you find. For more info, see discussions PRs #504, #505 and #508. Thanks to @andfoy, @rgommers and @FrancescAlted for the work. * Fix imaginary evaluation in the form of `1.1e1j`. This was previously not supported and would raise an error. Thanks to @27rabbitlt for the fix. * The test suite has been modernized to use `pytest` instead of `unittest`. This should make it easier to run the tests and contribute to the project. * Python 3.10 is now the minimum supported version. What's Numexpr? --- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? - The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience - Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3//lists/numpy-discussion.python.org Member address: arch...@mail-archive.com
