Re: [DISCUSSION] Ignite 3.0 and to be removed list

[Dmitry Melnichuk]

Dmitry,
As a Python thin client developer, I think that separate repository is
a truly great idea!
On Tue, 2019-06-18 at 21:29 +0300, Dmitriy Pavlov wrote:
> - Move to separate repositories: thin clients (at least non-Java
> 
> > ones)

Re: [DISCUSSION] Ignite 3.0 and to be removed list

[Dmitry Melnichuk]

Hello Igniters,

I'd like to add my ¢2 considering Python thin client.

I think we should abandon Python 3.4, which was a precondition from
Ignite community when I started to work on `pyignite` a good year ago,
and update the minimum requirement to at least Python 3.6, or, better
yet, 3.7.

Reasons to do so:

1. Python 3.4 has reached its EOL this Spring. The final build was
released: March 18, 2019. PIP, setuptools, PyCharm are already dropped
their support for Python 3.4. In other words, bits have started to rot.

2. Python 3.4 builds against OpenSSL 1.0, while 3.5+ − OpenSSL 1.1. If
we intend to support TLS 1.3 encryption in Python thin client, we
should move on from Python 3.4.

3. The most important part for me as a developer: positive changes in
the language itself. To name only a few:

- `dict` and `OrderedDict` types was merged within one highly-optimized 
C implementation, 

- `typing` module (type annotations support) was built into the
language and significantly improved,

- string interpolation (so-called “f-strings”) and data classes was
introduced in 3.6.

If there is at least one good reason to support Python 3.4, I would
like to learn about it. Otherwise, the struggle for supporting the
outdated language version seems pointless to me.

On Mon, 2019-06-17 at 15:18 +0300, Alexey Goncharuk wrote:
> Igniters,
> 
> Even though we are still planning the Ignite 2.8 release, I would
> like to
> kick-off a discussion related to Ignite 3.0, because the efforts for
> AI 3.0
> will be significantly larger than for AI 2.8, better to start early.
> 
> As a first step, I would like to discuss the list of things to be
> removed
> in Ignite 3.0 (partially this thread is inspired by Denis Magda's
> IGFS
> removal thread). I've separated all to-be-removed points from
> existing
> Ignite 3.0 wishlist [1] to a dedicated block and also added a few
> more
> things that look right to be dropped.
> 
> Please share your thoughts, probably, there are more outdated things
> we
> need to add to the wishlist.
> 
> As a side question: I think it makes sense to create tickets for such
> improvements, how do we track them. Will the 3.0 version suffice or
> should
> we add a separate label?
> 
> --AG

[jira] [Created] (IGNITE-11897) Use `bytearray` as a Python type for Ignite `ByteArray`

[Dmitry Melnichuk (JIRA)]

Dmitry Melnichuk created IGNITE-11897:
-

 Summary: Use `bytearray` as a Python type for Ignite `ByteArray`
 Key: IGNITE-11897
 URL: https://issues.apache.org/jira/browse/IGNITE-11897
 Project: Ignite
  Issue Type: Improvement
  Components: thin client
Reporter: Dmitry Melnichuk
Assignee: Dmitry Melnichuk


This optimization will allow client to read and write `ByteArray` values 
without iterating over single bytes, thereby improving performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Best effort affinity: NPE on topology version check

[Dmitry Melnichuk]

Igor,

Thank you! It helped. I adapted my code to the new protocol, so my
tests are passing.

On Tue, 2019-02-19 at 13:22 +0300, Igor Sapego wrote:
> This is my branch. I fixed an issue so re-merge and check if it
> helps.
> 
> Best Regards,
> Igor
> 
> 
> On Tue, Feb 19, 2019 at 8:14 AM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
> 
> > Igniters,
> > 
> > I am currently trying to teach my thin client (pyignite) a new
> > trick
> > called “Best effort affinity”.
> > 
> > I have been told that this feature is being implemented in the
> > private
> > branch [1], so I checked it out, built it, and gave it a go.
> > 
> > The handshake goes as expected, I have actually managed to read the
> > node UUID. But on any subsequent request the server disconnects and
> > shows the following on its console:
> > 
> > [ERROR][client-connector-#60][ClientListenerNioListener] Failed to
> > process client request [
> > 
> > req=o.a.i.i.processors.platform.client.cache.ClientCacheGetOrCreateWithNameRequest@43e45ce3
> > ]
> > java.lang.NullPointerException
> > 
> > The full dump is in attachment.
> > 
> > Since I am not authorized to create an issue in the repository I
> > was
> > using, I have to address my question to this list. If anybody
> > involved
> > in server-side implementation of best effort affinity is reading
> > this,
> > please tell me am I doing something wrong, or it is an actual bug?
> > How
> > can this issue be resolved?
> > 
> > [1] https://github.com/gridgain/apache-ignite/tree/ignite-iep23
> > 

Best effort affinity: NPE on topology version check

[Dmitry Melnichuk]

Igniters,

I am currently trying to teach my thin client (pyignite) a new trick
called “Best effort affinity”.

I have been told that this feature is being implemented in the private
branch [1], so I checked it out, built it, and gave it a go.

The handshake goes as expected, I have actually managed to read the
node UUID. But on any subsequent request the server disconnects and
shows the following on its console:

[ERROR][client-connector-#60][ClientListenerNioListener] Failed to
process client request [
req=o.a.i.i.processors.platform.client.cache.ClientCacheGetOrCreateWithNameRequest@43e45ce3
]
java.lang.NullPointerException

The full dump is in attachment.

Since I am not authorized to create an issue in the repository I was
using, I have to address my question to this list. If anybody involved
in server-side implementation of best effort affinity is reading this,
please tell me am I doing something wrong, or it is an actual bug? How
can this issue be resolved?

[1] https://github.com/gridgain/apache-ignite/tree/ignite-iep23
[18-Feb-2019 18:53:10][ERROR][client-connector-#60][ClientListenerNioListener] 
Failed to process client request 
[req=o.a.i.i.processors.platform.client.cache.ClientCacheGetOrCreateWithNameRequest@43e45ce3]
java.lang.NullPointerException
at 
org.apache.ignite.internal.processors.platform.client.ClientConnectionContext.checkAffinityTopologyVersion(ClientConnectionContext.java:200)
at 
org.apache.ignite.internal.processors.platform.client.ClientResponse.encode(ClientResponse.java:111)
at 
org.apache.ignite.internal.processors.platform.client.ClientMessageParser.encode(ClientMessageParser.java:403)
at 
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:198)
at 
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:48)
at 
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at 
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at 
org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at 
org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Re: Binary clients: fallback to the previous versions of the protocol

[Dmitry Melnichuk]

Igor,

I am sorry it took me a while to fully understand your reasoning.

“Update user software first, then update the server” approach still
looks somewhat weird to me (I think of Let's Encrypt client as an
example of “normal” approach in Python world), but since this approach
is vivid, I just have to take it into account, so I must agree with
you.

I just want to reiterate on one downside of such multi-protocol client,
that was not yet addressed (not in Jira tasks or in docs, at least).

Imagine a coder wrote a program with the latest client, using a feature
available only in latest binary protocol. When the coder tests his
program against the latest Ignite cluster, the program works perfectly.

But then the end user runs the program against the previous version of
the server, which client is still backwards-compatible with, the
program runs, but at some point it tries to use the latest feature of
the binary protocol and fails with some cryptic message. The end user
is clueless, so as the coder.

To avoid such a case, we must include an explicit parameter in our
client's initialization method, that would set the desired protocol
version(s) the user application is designed to work with. This
parameter should be explicit, i.e. not have a default value, since it
just will be useless the other way. And yes, this parameter renders all
the software built with previous client versions incompatible with the
new client.

I think this problem concerns not only the Python client, but all the
thin clients. What do you think?

On Wed, 2019-02-13 at 13:45 +0300, Igor Sapego wrote:
> The approach you suggest looks to me pretty much the same as
> installing a new version of client software in C++ or Java. The issue
> here that we break existing installed software and require for user
> to update software in order to have ability to connect to a server.
> Just imagine that application which made with thin client is not used
> by a developer that knows how to use pip and all the stuff, but
> someone with another background. Imagine, you have thousands of such
> users. And now imagine, you want to update your servers.
> 
> Best Regards,
> Igor
> 
> 
> On Tue, Feb 12, 2019 at 8:51 PM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
> 
> > Igor,
> > 
> > Thank you for your explanation. I think the matter begins to clear
> > up
> > for me now.
> > 
> > The backward compatibility issue you described can not be applied
> > to
> > Python, because Python applications, unlike Java ones, do not have
> > to
> > be built. They rely on package manager (pip, conda, et c.) to run
> > anywhere, including production.
> > 
> > At the stage of deployment, the package manager collects
> > dependencies
> > using a specially crafted response file, often called
> > `requirements.txt`.
> > 
> > For example, to ensure that their application will work with the
> > current _and_ future minor versions of pyignite, the user may
> > include a
> > line in their `requirements.txt` file:
> > 
> > pyignite < x
> > 
> > where x is a next major version number. In compliance with semantic
> > versioning, the line is basically says: “Use the latest available
> > version, that is earlier than x”.
> > 
> > When upgrading Ignite server, system administrator or devops
> > engineer
> > must also update or recreate the app's environment, or update OS-
> > level
> > packages, or redeploy the app using Docker − the exact procedure
> > may
> > vary, but in any case it should be completely standard − to deliver
> > the
> > latest suitable dependencies.
> > 
> > And then the same app connects to a latest Ignite server.
> > 
> > Here is more about how pip understands versions:
> > 
> > https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers
> > 
> > What we really need to do for this to work seamlessly, is to
> > establish
> > the clear relation between products' versions. Regretfully, I have
> > not
> > done this before; just did not expect for this issue to come up. I
> > think it would be best for pyignite major and minor to be set
> > according
> > to the Ignite binary protocol versions, i.e. pyignite 1.2.z handles
> > Ignite binary protocol v1.2, and so on. But that is another matter.
> > 
> > On Tue, 2019-02-12 at 13:39 +0300, Igor Sapego wrote:
> > > Hi,
> > > 
> > > Well, this approach was used for quite some time in ODBC and
> > > JDBC,
> > > so thin client has just inherited it, as we have the same port
> > > and
> > > the
> > > same handshake messa

[jira] [Created] (IGNITE-11303) Python thin client: best effort affinity

[Dmitry Melnichuk (JIRA)]

Dmitry Melnichuk created IGNITE-11303:
-

 Summary: Python thin client: best effort affinity
 Key: IGNITE-11303
 URL: https://issues.apache.org/jira/browse/IGNITE-11303
 Project: Ignite
  Issue Type: Task
  Components: thin client
Reporter: Dmitry Melnichuk
Assignee: Dmitry Melnichuk


The goal is to implement 
[IEP-23|https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients]
 using background thread (`threading` module).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Python examples are missing in Ignite 2.7

[Dmitry Melnichuk]

We just browse through the contents of the latest Ignite bundles [1].

Source bundle [2] contains the examples all right; they are in this
folder:

/apache-ignite-2.7.0-src/modules/platforms/python/examples/

Binary bundle [3] do not contain examples. I'm not totally sure, but
the reasoning I gave earlier might apply to binary bundle too.

Can anyone who's aware of the bundling procedure/scripts chime in,
please?

[1] https://ignite.apache.org/download.cgi

[2] 
http://mirrors.standaloneinstaller.com/apache//ignite/2.7.0/apache-ignite-2.7.0-src.zip

[3] 
http://mirrors.standaloneinstaller.com/apache//ignite/2.7.0/apache-ignite-2.7.0-bin.zip

On Wed, 2019-02-13 at 12:49 +1000, Dmitry Melnichuk wrote:
> Denis,
> 
> If by “release procedure” you mean the contents of the PyPI package,
> then it is not a bug, but a deliberate decision, that was documented
> in
> README [1]: 
> 
> > Installation
> > 
> > - for end user
> > 
> > If you only want to use the pyignite module in your project, do:
> > 
> > $ pip install pyignite
> > 
> > -- for developer
> > 
> > If you want to run tests, examples or build documentation, clone
> > the
> > whole repository…
> 
> The reasoning:
> 
> 1. The examples do not have much value by themselves. They are useful
> only in conjunction with the documentation. If we do not ship the
> documentation via PyPI, then we should not ship the examples either.
> 
> 2. In production environment, the extra packaged stuff like examples
> will be just a waste of space.
> 
> 3. Most Python libraries and frameworks I know of, e.g. Django or
> Scrapy, use the same approach: they have examples and test apps in
> the
> repository and reference them through their docs, but do not ship
> them 
> via PyPI.
> 
> [1] 
> https://github.com/apache/ignite/tree/master/modules/platforms/python#installation
> 
> On Tue, 2019-02-12 at 16:23 -0800, Denis Magda wrote:
> > Igniters,
> > 
> > Seems python examples were not added to the release bundle?
> > https://apacheignite.readme.io/v2.7/docs/python-thin-client#section-running-an-example
> > 
> > There is no "examples" folder for Python. Any flaws in the release
> > procedure?
> > 
> > -
> > Denis

Re: Python examples are missing in Ignite 2.7

[Dmitry Melnichuk]

Denis,

If by “release procedure” you mean the contents of the PyPI package,
then it is not a bug, but a deliberate decision, that was documented in
README [1]: 

> Installation
> 
> - for end user
> 
> If you only want to use the pyignite module in your project, do:
> 
> $ pip install pyignite
> 
> -- for developer
> 
> If you want to run tests, examples or build documentation, clone the
> whole repository…

The reasoning:

1. The examples do not have much value by themselves. They are useful
only in conjunction with the documentation. If we do not ship the
documentation via PyPI, then we should not ship the examples either.

2. In production environment, the extra packaged stuff like examples
will be just a waste of space.

3. Most Python libraries and frameworks I know of, e.g. Django or
Scrapy, use the same approach: they have examples and test apps in the
repository and reference them through their docs, but do not ship them 
via PyPI.

[1] 
https://github.com/apache/ignite/tree/master/modules/platforms/python#installation

On Tue, 2019-02-12 at 16:23 -0800, Denis Magda wrote:
> Igniters,
> 
> Seems python examples were not added to the release bundle?
> https://apacheignite.readme.io/v2.7/docs/python-thin-client#section-running-an-example
> 
> There is no "examples" folder for Python. Any flaws in the release
> procedure?
> 
> -
> Denis

Re: Binary clients: fallback to the previous versions of the protocol

[Dmitry Melnichuk]

Igor,

Thank you for your explanation. I think the matter begins to clear up
for me now.

The backward compatibility issue you described can not be applied to
Python, because Python applications, unlike Java ones, do not have to
be built. They rely on package manager (pip, conda, et c.) to run
anywhere, including production.

At the stage of deployment, the package manager collects dependencies
using a specially crafted response file, often called
`requirements.txt`.

For example, to ensure that their application will work with the
current _and_ future minor versions of pyignite, the user may include a
line in their `requirements.txt` file:

pyignite < x

where x is a next major version number. In compliance with semantic
versioning, the line is basically says: “Use the latest available
version, that is earlier than x”.

When upgrading Ignite server, system administrator or devops engineer
must also update or recreate the app's environment, or update OS-level
packages, or redeploy the app using Docker − the exact procedure may
vary, but in any case it should be completely standard − to deliver the
latest suitable dependencies.

And then the same app connects to a latest Ignite server.

Here is more about how pip understands versions:

https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers

What we really need to do for this to work seamlessly, is to establish
the clear relation between products' versions. Regretfully, I have not
done this before; just did not expect for this issue to come up. I
think it would be best for pyignite major and minor to be set according
to the Ignite binary protocol versions, i.e. pyignite 1.2.z handles
Ignite binary protocol v1.2, and so on. But that is another matter.

On Tue, 2019-02-12 at 13:39 +0300, Igor Sapego wrote:
> Hi,
> 
> Well, this approach was used for quite some time in ODBC and JDBC,
> so thin client has just inherited it, as we have the same port and
> the
> same handshake message header. Maybe that's why you could not find
> any mention of versioning in thin client context.
> 
> 3. Let's start from this point as this is the root of the issue.
> Preconditions
> are simple - the backward compatibility. We should guarantee that
> user
> can use any version of thin client with any version of server while
> they
> share the same major version. This can be important, if someone have
> user applications built with thin client and they don't want to break
> their
> user applications, when they just want to update servers. What is
> appropriate here is to give user some kind of warning that they use
> outdated client or server.
> 
> Can you explain how your approach is going to work in the example
> above? I'm not quite sure I understand you correctly.
> 
> 1. The API for this can be as simple as a set of allowed version in
> a configuration for both server and client. Maybe we should discuss
> such
> kind of API as it can be useful for example in terms of security.
> 
> 2. Complexity is the price developers have to pay for better user
> experience.
> Nothing new here. Software is not designed for developers, it's
> designed
> for a user.
> 
> Best Regards,
> Igor
> 
> 
> On Tue, Feb 12, 2019 at 11:56 AM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
> 
> > Hello fellow igniters!
> > 
> > I recently started reconnaissance before the
> > implementation of IEP-23 [1] in Python thin client. I think it is
> > an
> > interesting and much promising feature. As I understand, the
> > changes
> > associated with this feature will result in a new version of the
> > binary
> > protocol.
> > 
> > What bothers me is not the changes required by the
> > implementation itself, but the upcoming changes in the way that the
> > fallback to the previous versions of the protocol is going to be
> > handled.
> > 
> > I have not found a open discussion on this topic neither on
> > this mailing list, nor in Jira. If such a discussion took place,
> > please
> > inform me with a link. Based on what I have heard so far, I made
> > the
> > following conclusions:
> > 
> > 1) the client is now expected to handle multiple
> > binary protocol versions;
> > 2) since the protocol is implemented the way
> > the client first reports its version to the server on handshake,
> > and
> > not the other way, the client now must try to connect to the server
> > using its preferred version of the protocol, and in case of an
> > error,
> > reconnect with another version.
> > 
> > These changes are raised some concerns
> > for me, for example:
> > 
> > 1) the version negotiation mech

Binary clients: fallback to the previous versions of the protocol

[Dmitry Melnichuk]

Hello fellow igniters!

I recently started reconnaissance before the
implementation of IEP-23 [1] in Python thin client. I think it is an
interesting and much promising feature. As I understand, the changes
associated with this feature will result in a new version of the binary
protocol.

What bothers me is not the changes required by the
implementation itself, but the upcoming changes in the way that the
fallback to the previous versions of the protocol is going to be
handled.

I have not found a open discussion on this topic neither on
this mailing list, nor in Jira. If such a discussion took place, please
inform me with a link. Based on what I have heard so far, I made the
following conclusions:

1) the client is now expected to handle multiple
binary protocol versions;
2) since the protocol is implemented the way
the client first reports its version to the server on handshake, and
not the other way, the client now must try to connect to the server
using its preferred version of the protocol, and in case of an error,
reconnect with another version.

These changes are raised some concerns
for me, for example:

1) the version negotiation mechanism implemented on
client can not be made transparent to the end user. We must give the
user an option, so that they could choose from the entire set of
versions supported by the client, a subset of the versions their
application is designed to work with. That will complicate the user
API;

2) the complexity of the client will also be increased, especially
considering unit testing. Certain tests will require a certain version
of Ignite server, so they can no longer be implemented environment-
agnostically, leading to a weird mix on unit and integration levels;

3)
I do not see, or not aware of, the preconditions to such changes. I
think the number of potential applications that can benefit from multi-
protocol client is quite low. Usually all the hassle of version
matching lie on the deployment level, above the app level. Please prove
me wrong on this subject.

Instead of supporting multiple binary
protocols inside one client package, I would strongly prefer to support
each version of the protocol separately, in the corresponding branch of
the package repository. It would still be possible for the end user to
use multiple protocols in one app, but with certain code-level and
deployment-level efforts combined.

Again, I most probably lack awareness
on the subject, and therefore my conclusions may be premature, but
anyway, let us discuss. I will appreciate any bit of knowledge from the
community.

[1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-
23:+Best+Effort+Affinity+for+thin+clients

Re: [RESULT] [VOTE] Apache Ignite 2.7.0 Release (RC2)

[Dmitry Melnichuk]

Nikolay, Petr,

I have already registered the package `pyignite` on PyPI[1]. The person 
who is going to take the responsibility of maintaining it should create 
an account on PyPI and mail me in private, so that I can grant them the 
necessary rights. They also must install twine[3].


The process of packaging is well described in the packaging tutorial[2]. 
In the nutshell, the maintainer must do the following:


1. Clone/pull the sources from the git repository,
2. Enter the directory in which the `setup.py` is resides (“the setup 
directory”), in our case it is `modules/platforms/python`.
3. Create the packages with the command `python3 setup.py sdist 
bdist_wheel`. The packages will be created in 
`modules/platforms/python/dist` folder.

4. Upload packages with twine: `twine upload dist/*`.

It is very useful to have a dedicated Python virtual environment 
prepared to perform steps 3-4. Just do an editable install of `pyignite` 
into that environment from the setup directory: `pip3 install -e .` You 
can also install twine (`pip install twine`) in it.


Consider also making a `.pypirc` file to save time on logging in to 
PyPI. Newest version of `twine` is said to support keyrings on Linux and 
Mac, but I have not tried this yet.


[1] https://pypi.org/project/pyignite/
[2] https://packaging.python.org/tutorials/packaging-projects/
[3] https://twine.readthedocs.io/en/latest/

Some other notes on PyPI and versioning.

- The package version is located in the `setup.py`, it is a `version` 
argument of the `setuptools.setup()` function. Editing the `setup.py` is 
the only way to set the package version.


- You absolutely can not replace a package in PyPI (hijacking 
prevention). If you have published the package by mistake, all you can 
do is delete the unwanted package, increment the version counter in 
`setup.py`, and try again.


- If you upload the package through the web interface of PyPI (without 
twine), the package description will be garbled. Web interface does not 
support markdown.


Anyway, I would like to join in the congratulations on successful 
release. Kudos to the team.


On 12/10/18 4:49 PM, Nikolay Izhikov wrote:

Hello, Roman.

Do we have release procedure for this pacakage?
Let's develop one.

Thin client developers, please,let me know, which artifacts should be uploaded 
to which package manager system.

Re: Thin clients all in one

[Dmitry Melnichuk]

Stepan,

AFAIK Map type did always behave correctly on client side, as it does 
now. This is a corresponding piece of my test suite:


```
def test_put_get_map(client):

cache = client.get_or_create_cache('test_map_cache')

cache.put(
'test_map',
(
MapObject.HASH_MAP,
{
(123, IntObject): 'test_data',
456: ((1, [456, 'inner_test_string', 789]), 
CollectionObject),

'test_key': 32.4,
}
),
value_hint=MapObject
)
value = cache.get('test_map')
assert value == (MapObject.HASH_MAP, {
123: 'test_data',
456: (1, [456, 'inner_test_string', 789]),
'test_key': 32.4,
})

```

Or is there another, more specific problem with maps?

Dmitry

On 11/25/18 3:56 AM, Stepan Pilschikov wrote:

Dmitry,

Great, checked, now all things woks well
Hope that Igor made review for this PR

But what about Maps? Looks like different ticket? or it can be done in same
ticket scope?

пт, 23 нояб. 2018 г. в 23:58, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan,

Sorry, I forgot to update from upstream prior to start working on this
issue, and thus brought a regression. My bad. Just merged with the
latest master. Please, check it out again.

Dmitry

On 11/24/18 1:37 AM, Stepan Pilschikov wrote:

Dmitry,

Iv checked and its actually work
But a specially in this branch i found another bug
Please look at my last comment:


https://issues.apache.org/jira/browse/IGNITE-10358?focusedCommentId=16697285&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16697285


пт, 23 нояб. 2018 г. в 01:21, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan,

Thank you for your great job in evaluating Python thin client, as well
as other thin clients.

There was indeed a bug in Python client regarding the handling of type
hints in Collection type. I created a fix and did a PR under
IGNITE-10358 task, but the same PR is also fixes the problem in
IGNITE-10230 task.

As of handling the type mapping in gists you provided, I left comments
on both tasks.

Dmitry

On 11/21/18 6:37 PM, Stepan Pilschikov wrote:

Dmitry, Alexey

Thank you for help, this answers help me a lot with understanding how
clients are work

Not so long time ago i met problem which is have expected behavior, but

its

may broke some workflows in future for some users

Its all about not specified data types in collections and map's
All description and examples in
https://issues.apache.org/jira/browse/IGNITE-10358

Dmitry, can you have a quick look at it and maybe in future somehow fix

it?


пт, 26 окт. 2018 г. в 19:05, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan!

TL/DR: what you got with Python client in your gist is an intended
behavior.

Explanation: As per docs, Object array contains of type ID (which is
defaults to -1) and an array of objects.






https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array


Your gist might be fixed accordingly:

```
from pyignite import Client
from pyignite.datatypes import *

OBJECT_ARRAY_TYPE_ID = -1
OBJECT_ARRAY_CONTENTS = [1, 2]

client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
cache.put(
1,
(OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
key_hint=IntObject,
value_hint=ObjectArrayObject,
)

# Python  output: print(cache.get(1))
# (-1, [1, 2])
```

The situation is similar with Map and Collection, they have types.

Types

and type IDs are mostly useless in Python, but I left them for
interoperability reasons. If you think I should kick them out, just

let

me know.

The usage of these 3 data types is documented and tested. You can

refer

to the table in “Data types” section:






https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html


The tests are here:






https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124


On 10/26/18 11:57 PM, Stepan Pilschikov wrote:

Hi, everyone

Create new thread to centralize cross compatibility and others common
problems between thin clients

Tying to use Object array to exchange different data between JS, PHP

and

Python thin clients

JS and PHP simply can't put any type of arrays
Python can put data, but if you take it, data would be completely
different, maybe during this put process data is changed

Code and output:
PHP -

https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a

JS -

https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d

Python -
https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f

I'm tried different data types (string, double, float, complex

objects,

just random objec

Re: Thin clients all in one

[Dmitry Melnichuk]

Stepan,

Sorry, I forgot to update from upstream prior to start working on this 
issue, and thus brought a regression. My bad. Just merged with the 
latest master. Please, check it out again.


Dmitry

On 11/24/18 1:37 AM, Stepan Pilschikov wrote:

Dmitry,

Iv checked and its actually work
But a specially in this branch i found another bug
Please look at my last comment:
https://issues.apache.org/jira/browse/IGNITE-10358?focusedCommentId=16697285&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16697285

пт, 23 нояб. 2018 г. в 01:21, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan,

Thank you for your great job in evaluating Python thin client, as well
as other thin clients.

There was indeed a bug in Python client regarding the handling of type
hints in Collection type. I created a fix and did a PR under
IGNITE-10358 task, but the same PR is also fixes the problem in
IGNITE-10230 task.

As of handling the type mapping in gists you provided, I left comments
on both tasks.

Dmitry

On 11/21/18 6:37 PM, Stepan Pilschikov wrote:

Dmitry, Alexey

Thank you for help, this answers help me a lot with understanding how
clients are work

Not so long time ago i met problem which is have expected behavior, but

its

may broke some workflows in future for some users

Its all about not specified data types in collections and map's
All description and examples in
https://issues.apache.org/jira/browse/IGNITE-10358

Dmitry, can you have a quick look at it and maybe in future somehow fix

it?


пт, 26 окт. 2018 г. в 19:05, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan!

TL/DR: what you got with Python client in your gist is an intended
behavior.

Explanation: As per docs, Object array contains of type ID (which is
defaults to -1) and an array of objects.




https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array


Your gist might be fixed accordingly:

```
from pyignite import Client
from pyignite.datatypes import *

OBJECT_ARRAY_TYPE_ID = -1
OBJECT_ARRAY_CONTENTS = [1, 2]

client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
cache.put(
   1,
   (OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
   key_hint=IntObject,
   value_hint=ObjectArrayObject,
)

# Python  output: print(cache.get(1))
# (-1, [1, 2])
```

The situation is similar with Map and Collection, they have types. Types
and type IDs are mostly useless in Python, but I left them for
interoperability reasons. If you think I should kick them out, just let
me know.

The usage of these 3 data types is documented and tested. You can refer
to the table in “Data types” section:




https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html


The tests are here:




https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124


On 10/26/18 11:57 PM, Stepan Pilschikov wrote:

Hi, everyone

Create new thread to centralize cross compatibility and others common
problems between thin clients

Tying to use Object array to exchange different data between JS, PHP

and

Python thin clients

JS and PHP simply can't put any type of arrays
Python can put data, but if you take it, data would be completely
different, maybe during this put process data is changed

Code and output:
PHP -

https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a

JS -

https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d

Python -
https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f

I'm tried different data types (string, double, float, complex objects,
just random objects, char, byte, Date), still

How, from my perspective, it should works:
put array of any type and then get this array.
Example: put [1,2,3] -> get [1,2,3] or put [new Date(), new Date()] ->
get [[Date object], [Date object]] (like in java thin client)

Looks like bug in all clients, what you think?

I wanted to try Collections, but i think this type have same problem

Re: Thin clients all in one

[Dmitry Melnichuk]

Stepan,

Thank you for your great job in evaluating Python thin client, as well 
as other thin clients.


There was indeed a bug in Python client regarding the handling of type 
hints in Collection type. I created a fix and did a PR under 
IGNITE-10358 task, but the same PR is also fixes the problem in 
IGNITE-10230 task.


As of handling the type mapping in gists you provided, I left comments 
on both tasks.


Dmitry

On 11/21/18 6:37 PM, Stepan Pilschikov wrote:

Dmitry, Alexey

Thank you for help, this answers help me a lot with understanding how
clients are work

Not so long time ago i met problem which is have expected behavior, but its
may broke some workflows in future for some users

Its all about not specified data types in collections and map's
All description and examples in
https://issues.apache.org/jira/browse/IGNITE-10358

Dmitry, can you have a quick look at it and maybe in future somehow fix it?

пт, 26 окт. 2018 г. в 19:05, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan!

TL/DR: what you got with Python client in your gist is an intended
behavior.

Explanation: As per docs, Object array contains of type ID (which is
defaults to -1) and an array of objects.


https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array

Your gist might be fixed accordingly:

```
from pyignite import Client
from pyignite.datatypes import *

OBJECT_ARRAY_TYPE_ID = -1
OBJECT_ARRAY_CONTENTS = [1, 2]

client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
cache.put(
  1,
  (OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
  key_hint=IntObject,
  value_hint=ObjectArrayObject,
)

# Python  output: print(cache.get(1))
# (-1, [1, 2])
```

The situation is similar with Map and Collection, they have types. Types
and type IDs are mostly useless in Python, but I left them for
interoperability reasons. If you think I should kick them out, just let
me know.

The usage of these 3 data types is documented and tested. You can refer
to the table in “Data types” section:


https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html

The tests are here:


https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124

On 10/26/18 11:57 PM, Stepan Pilschikov wrote:

Hi, everyone

Create new thread to centralize cross compatibility and others common
problems between thin clients

Tying to use Object array to exchange different data between JS, PHP and
Python thin clients

JS and PHP simply can't put any type of arrays
Python can put data, but if you take it, data would be completely
different, maybe during this put process data is changed

Code and output:
PHP -

https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a

JS -

https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d

Python -
https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f

I'm tried different data types (string, double, float, complex objects,
just random objects, char, byte, Date), still

How, from my perspective, it should works:
put array of any type and then get this array.
Example: put [1,2,3] -> get [1,2,3] or put [new Date(), new Date()] ->
get [[Date object], [Date object]] (like in java thin client)

Looks like bug in all clients, what you think?

I wanted to try Collections, but i think this type have same problem

Re: Thin clients all in one

[Dmitry Melnichuk]

Stepan!

TL/DR: what you got with Python client in your gist is an intended behavior.

Explanation: As per docs, Object array contains of type ID (which is 
defaults to -1) and an array of objects.


https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array

Your gist might be fixed accordingly:

```
from pyignite import Client
from pyignite.datatypes import *

OBJECT_ARRAY_TYPE_ID = -1
OBJECT_ARRAY_CONTENTS = [1, 2]

client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
cache.put(
1,
(OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
key_hint=IntObject,
value_hint=ObjectArrayObject,
)

# Python  output: print(cache.get(1))
# (-1, [1, 2])
```

The situation is similar with Map and Collection, they have types. Types 
and type IDs are mostly useless in Python, but I left them for 
interoperability reasons. If you think I should kick them out, just let 
me know.


The usage of these 3 data types is documented and tested. You can refer 
to the table in “Data types” section:


https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html

The tests are here:

https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124

On 10/26/18 11:57 PM, Stepan Pilschikov wrote:

Hi, everyone

Create new thread to centralize cross compatibility and others common 
problems between thin clients


Tying to use Object array to exchange different data between JS, PHP and 
Python thin clients


JS and PHP simply can't put any type of arrays
Python can put data, but if you take it, data would be completely 
different, maybe during this put process data is changed


Code and output:
PHP - https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a
JS - https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d
Python - 
https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f


I'm tried different data types (string, double, float, complex objects, 
just random objects, char, byte, Date), still


How, from my perspective, it should works:
put array of any type and then get this array.
Example: put [1,2,3] -> get [1,2,3] or put [new Date(), new Date()] -> 
get [[Date object], [Date object]] (like in java thin client)


Looks like bug in all clients, what you think?

I wanted to try Collections, but i think this type have same problem

Re: Python thin client

[Dmitry Melnichuk]

Stepan,

My bad. I got the Decimal part of Ignite thin client protocol 
specification totally wrong.


Just submitted a patch tested against Java thin client. Please evaluate.

On 10/20/18 2:00 AM, Stepan Pilschikov wrote:

Dmitry,

One more thing
Discover that Decimal data type can't be getting by others clients

Make ticket with description:
https://issues.apache.org/jira/browse/IGNITE-9950

Please, can you help to investigate this problem?

ср, 17 окт. 2018 г. в 23:14, Stepan Pilschikov :


Dmitry,

Great, now client works properly on all OS
Thanks

I hope PR will be merged soon

Igor, can please help with that?

ср, 17 окт. 2018 г. в 20:48, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Stepan,

Sadly, it was a C `long` data type size mismatch between Windows and
other OSes. I investigated this issue and offered a pull request.

https://github.com/apache/ignite/pull/5017

On 10/17/18 6:29 PM, Stepan Pilschikov wrote:

Dmitry,

I've trying to use python thin client with Ignite 2.7 it dosn't work
Please help to investigate
Problem reproduced on several others machines

Issue:
https://issues.apache.org/jira/browse/IGNITE-9908

Re: Python thin client

[Dmitry Melnichuk]

Stepan,

Sadly, it was a C `long` data type size mismatch between Windows and 
other OSes. I investigated this issue and offered a pull request.


https://github.com/apache/ignite/pull/5017

On 10/17/18 6:29 PM, Stepan Pilschikov wrote:

Dmitry,

I've trying to use python thin client with Ignite 2.7 it dosn't work
Please help to investigate
Problem reproduced on several others machines

Issue:
https://issues.apache.org/jira/browse/IGNITE-9908

Re: Python thin client

[Dmitry Melnichuk]

Igor,

I do not have access to edit wiki pages, so please fill it.

All the features are supported, except for:

- Query API: ScanQuery with filter,
- Binary API: Get object in BinaryObject form, BinaryObject building,
- Additional Features: Best effort affinity,
- Type registration: Enum registration, Type name registration.

On 10/15/18 9:17 PM, Igor Sapego wrote:

Dmitry,

Since Python thin client is now in master, can you please add it to our
"Thin clients features" wiki page, so we can track the feature parity
of our clients? Or just tell me which features are supported, so I can fill
it myself.

[1] -
https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features

Best Regards,
Igor

Re: Python thin client

[Dmitry Melnichuk]

Igor,

I fixed my files.

Dmitry

On 10/13/18 12:14 AM, Igor Sapego wrote:

Dmitry,

I've run the tests and it appears that there are a lot of files without
Apache
licenses in Python client: [1] (See Unapproved licenses list).

Can you either add headers or exclude file from the check, if you can not
add one? I.e. to this list: [2]

[1] -
https://ci.ignite.apache.org/repository/download/IgniteTests24Java8_LicensesHeaders/2069024:id/rat.zip%21/target/rat.txt
[2] - https://github.com/apache/ignite/blob/master/parent/pom.xml#L817

Best Regards,
Igor


On Thu, Oct 11, 2018 at 1:13 PM Igor Sapego  wrote:


Ok, I've filed a ticket [1] for the performance investigation and targeted
it to 2.8. Feel free to assign it to yourself.

I'm going to merge the ticket to master and ignite-2.7 branches by the
end of the week, if no one have any objections.

[1] - https://issues.apache.org/jira/browse/IGNITE-9850

Best Regards,
Igor


On Thu, Oct 11, 2018 at 12:18 PM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Stepan,

Thank you for answering my question and for the updated results.

I have ran the profiler (cProfile) against the benchmark code you
provided and have not found any particular bottleneck. One cycle took me
434 μs, which is the same order of magnitude as your result.

I will need more time to analyze the performance of my client and find a
way to improve it. But is it a real stopper/blocker for the client
itself? If there are no other issues, may we merge and release the
client? Plus submit a jira-improvement to further investigate the
performance of the Python platform.

Dmitry

On 10/11/18 4:19 PM, Степан Пильщиков wrote:

Dmitry,

pip install -e from latest sources

On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, <

dmitry.melnic...@nobitlost.com>

wrote:


Stepan!

Please tell me one thing about how you created the environment for your
benchmarks, namely: how did you install the Python client. Did you just
do `pip install pyignite`, or did you pull my working branch from the
downstream repository and did `pip install -e
ignite/modules/platforms/python`?

I highly recommend the latter, because PyPI version do not yet include
the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
These improvements can have a noticeable positive effect on speed.

Dmitry

On 10/10/18 11:06 PM, Степан Пильщиков wrote:

Dmitry,

Thanks for review

Agree with all suggestions.

Made and run new benches without any redundant operations (prints,

time

calculation), only with "put" in "while true" cycle (almost)
Case description, environment, results and code in ticket
https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)

Please review new metrics
Because picture is not changed so much, python still have performance

issues

Re: Python thin client

[Dmitry Melnichuk]

Stepan,

Thank you for answering my question and for the updated results.

I have ran the profiler (cProfile) against the benchmark code you 
provided and have not found any particular bottleneck. One cycle took me 
434 μs, which is the same order of magnitude as your result.


I will need more time to analyze the performance of my client and find a 
way to improve it. But is it a real stopper/blocker for the client 
itself? If there are no other issues, may we merge and release the 
client? Plus submit a jira-improvement to further investigate the 
performance of the Python platform.


Dmitry

On 10/11/18 4:19 PM, Степан Пильщиков wrote:

Dmitry,

pip install -e from latest sources

On Thu, 11 Oct 2018, 06:37 Dmitry Melnichuk, 
wrote:


Stepan!

Please tell me one thing about how you created the environment for your
benchmarks, namely: how did you install the Python client. Did you just
do `pip install pyignite`, or did you pull my working branch from the
downstream repository and did `pip install -e
ignite/modules/platforms/python`?

I highly recommend the latter, because PyPI version do not yet include
the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov.
These improvements can have a noticeable positive effect on speed.

Dmitry

On 10/10/18 11:06 PM, Степан Пильщиков wrote:

Dmitry,

Thanks for review

Agree with all suggestions.

Made and run new benches without any redundant operations (prints, time
calculation), only with "put" in "while true" cycle (almost)
Case description, environment, results and code in ticket
https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)

Please review new metrics
Because picture is not changed so much, python still have performance

issues

Re: Python thin client

[Dmitry Melnichuk]

Stepan!

Please tell me one thing about how you created the environment for your 
benchmarks, namely: how did you install the Python client. Did you just 
do `pip install pyignite`, or did you pull my working branch from the 
downstream repository and did `pip install -e 
ignite/modules/platforms/python`?


I highly recommend the latter, because PyPI version do not yet include 
the latest improvements I did thanks to Dmitriy “qvad” Sherstobitov. 
These improvements can have a noticeable positive effect on speed.


Dmitry

On 10/10/18 11:06 PM, Степан Пильщиков wrote:

Dmitry,

Thanks for review

Agree with all suggestions.

Made and run new benches without any redundant operations (prints, time
calculation), only with "put" in "while true" cycle (almost)
Case description, environment, results and code in ticket
https://issues.apache.org/jira/browse/IGNITE-9824 (last comment)

Please review new metrics
Because picture is not changed so much, python still have performance issues

Re: Python thin client

[Dmitry Melnichuk]

Stepan!

Can you please update the benchmarks based on my suggestions earlier in 
this thread: use a cleaner time profiling for the loop, remove 
unnecessary operations (string formatting, rounding operations), stick 
to the primitive int value for put operations (agree with Vladimir, 
let's keep it simple). And let me know the results.


On 10/10/18 5:46 PM, Vladimir Ozerov wrote:

Hi Dmitry,

I agree with your comments on benchmarking code. As more fair 
alternative we may measure time of loading N elements into the cache, so 
that it will be necessary to call time() only twice. However, provided 
that we have real network here, and according to numbers one PUT in 
Python client requires about 0.3 msec, I hardly can imagine that call to 
time() or round() may dominate in that response time. But I said, we can 
easily rule that out by slight benchmark rewrite.


As far as serialization, I would prefer to keep primitive objects at the 
moment, because AFAIK the purpose of the benchmark was to assess 
underlying infrastructure, looking for some obvious performance issues. 
Every platform have sockets which use more or less the same API of 
underlying OS. To the contrast, performance of serialization mechanics 
may differ widely between platforms depending on implementation. E.g. in 
Java we spend a lot of time on fine-grained tuning, applied a lot of 
speculative optimizations. Add to this JIT nature of Java, and you will 
hardly ever beat Java serialization engine from any interpreted 
language. So primitive data types make good sense to me.


At this point our goal is not make Python equally fast with other 
platforms, but rather to understand why is it slower than others. Ignite 
is a product which brings speed to end users. If they do no have speed, 
they will not use Ignite. So performance questions are always of great 
importance for us.


Vladimir.

On Wed, Oct 10, 2018 at 9:57 AM Dmitry Melnichuk 
mailto:dmitry.melnic...@nobitlost.com>> 
wrote:


Hi, Stepan!

I looked at the benchmark code and the overall methodics, discussed it
with fellow programmers, and came up with basically two remarks.

First of all, I think, the key for the good (or, at least,
unobjectionable) measurement is to isolate the object being measured
from the influence of the measurement tool. The usual pattern we use in
Python looks like:

```
from time import time


bench_start = time()

for _ in range(number_of_tests):
      do_test()

bench_end = time()

print('Performance is {} tests per second'.format(
      number_of_tests / (bench_end - bench_start)
))
```

I think, you got the idea: the measurement consists almost solely of
the
time taken by our subject function `do_test`. As few other code as
possible influence the result.

Now, let's take a look at your code:

https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb

Ideally, the `while` loop should include only `cache.put(last_key,
some_precomputed_value)`. But instead it also includes:

- a series of the `time()` calls, which could be mostly excluded from
the measured time, if the measurement done right; each call is probably
addresses the HPET device, or network time, or both,

- some floating-point calculations, including `round()`, which is
hardly
necessary,

- formatting and output of the intermediate result.

I suppose the measurement influence can be quite significant here, but
it is at least more or less constant for each test.

But if we look at the other benchmarks:

https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447
[Node.js]
https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80
[PHP]
https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce
[Java]

The extra code that influence the measurement is not equivalent across
all platforms. For example, PHP's `time()` is most probably lighter
than
Python `time()`, since it do not give out milliseconds and may address
RTC, not HPET. So the platform-to-platform comparison in your benchmark
does not look completely fair to me.

The second remark concerns not the measurement procedure, but the
object
being measured.

The only client operation being used is OP_CACHE_PUT with a payload
of a
primitive type. (BTW the type is `Long` in case of the Python client;
what about the other clients? Is it `Int`?) I afraid that such an
object, even being properly isolated from the measurement tool, would
show mostly the throughput of the underlying platform's sockets
implementation, not the performance of the client's code. To show the
potential of the thin client itself, more varied and fancy tasks
needed,
i. e. serialization/deserialization of the Complex objects.

But it depends on the goal of

Re: Python thin client

[Dmitry Melnichuk]

Hi, Stepan!

I looked at the benchmark code and the overall methodics, discussed it 
with fellow programmers, and came up with basically two remarks.


First of all, I think, the key for the good (or, at least, 
unobjectionable) measurement is to isolate the object being measured 
from the influence of the measurement tool. The usual pattern we use in 
Python looks like:


```
from time import time


bench_start = time()

for _ in range(number_of_tests):
do_test()

bench_end = time()

print('Performance is {} tests per second'.format(
number_of_tests / (bench_end - bench_start)
))
```

I think, you got the idea: the measurement consists almost solely of the 
time taken by our subject function `do_test`. As few other code as 
possible influence the result.


Now, let's take a look at your code:

https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb

Ideally, the `while` loop should include only `cache.put(last_key, 
some_precomputed_value)`. But instead it also includes:


- a series of the `time()` calls, which could be mostly excluded from 
the measured time, if the measurement done right; each call is probably 
addresses the HPET device, or network time, or both,


- some floating-point calculations, including `round()`, which is hardly 
necessary,


- formatting and output of the intermediate result.

I suppose the measurement influence can be quite significant here, but 
it is at least more or less constant for each test.


But if we look at the other benchmarks:

https://gist.github.com/pilshchikov/8a4bdb03a8304136c22c9bf7217ee447 
[Node.js]

https://gist.github.com/pilshchikov/b4351d78ad59e9cd923689c2e387bc80 [PHP]
https://gist.github.com/pilshchikov/08096c78b425e00166a2ffa2aa5f49ce [Java]

The extra code that influence the measurement is not equivalent across 
all platforms. For example, PHP's `time()` is most probably lighter than 
Python `time()`, since it do not give out milliseconds and may address 
RTC, not HPET. So the platform-to-platform comparison in your benchmark 
does not look completely fair to me.


The second remark concerns not the measurement procedure, but the object 
being measured.


The only client operation being used is OP_CACHE_PUT with a payload of a 
primitive type. (BTW the type is `Long` in case of the Python client; 
what about the other clients? Is it `Int`?) I afraid that such an 
object, even being properly isolated from the measurement tool, would 
show mostly the throughput of the underlying platform's sockets 
implementation, not the performance of the client's code. To show the 
potential of the thin client itself, more varied and fancy tasks needed, 
i. e. serialization/deserialization of the Complex objects.


But it depends on the goal of the benchmarking. If showing off the raw 
platform agility was intended, than this objection is removed.


Dmitry

On 10/9/18 10:50 PM, Stepan Pilshchikov wrote:

Hi, all

Tried to compare new thin client performance and made similar benchmarks for
each one
And result not so good for python

Ticket with results and bench code:
https://issues.apache.org/jira/browse/IGNITE-9824

Py code src:
https://gist.github.com/pilshchikov/8aff4e30d83f8bac20c5a4a9c3917abb

Dmitry, please review results and bench code, maybe somthing wrong or it's
expected numbers?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Python thin client installation instructions

[Dmitry Melnichuk]

Prachi,

I feel your struggle. It is easier for end user to perceive Python 2 and 
Python 3 as different languages, not as versions of one language. They 
usually installed alongside each other; their updates are handled 
separately. On most systems they have their respective shell commands: 
`python2` and `python3`.


Shell command `python` can be viewed as an alias of either `python2` or 
`python3`. I use an Arch Linux derivative, where `python` is `python3`. 
Most other GNU/Linux OSes use `python` as an alias of `python2`. I am 
not sure about MacOS. On Windows the latest Python distribution 
installed overrule PATH environment variables, so it impossible to 
predict the “default” Python version (2 or 3).


Luckily, virtualenv was introduced to leverage all these issues. It is 
able to handle multiple isolated Python environments, where the `python` 
command is set upon the creation of the environment, while the 
environment-specific package dependencies are handled transparently with 
pip.


But the use of pyignite should not be limited to virtualenv. There are 
many cases when the use of virtualenv is discouraged or even impossible. 
For example, when deploying Python app into an OS-level container or 
similar isolating environment, virtualenv would be just a useless 
overhead. There are also non-standard Python distributions (used mostly 
on Windows) that do not support virtualenv.


I am sorry, that users who are not proficient in Python can have so many 
problems with following my documentation. But still it seems obvious for 
me, that all the details of organizing user's own Python environment are 
out of pyignite documentation's scope. The only thing I can suggest to 
improve my documentation in this regard is putting a big bold foreword 
like this:


  It is assumed in this document that you know how to install
  and use Python 3 on your system. Please consult your OS manual pages
  or documentations of your specific Python 3 distribution regarding
  the details of organizing your Python 3 environment. The use of
  virualenv for development with pyignite is highly recommended.

But, frankly, I have not seen such disclaimers in the wild and not sure 
if it would be useful. It is very vague and do not cover any of the 
potential pitfalls.


I am sorry for giving such a lengthy explanation here, though I've been 
asked a very specific question. I understand you may not have time to 
invest in learning virtualenv. If so, you did everything right, just use 
`python3` command for launching examples:


```
$ python3 get_and_put.py
```

On 9/21/18 10:34 AM, Prachi Garg wrote:

Hi Dmitry,

Sorry, I am not familiar with Python.

So there are more issues...

1. The version on my mac remains 2.7.10 even though I tried to link to 
the new version.


~$ python --version
Python 2.7.10

~$ brew unlink python && brew link --overwrite python3
Unlinking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks 
removed
Linking /usr/local/Cellar/python/3.7.0. <http://3.7.0.>.. 25 symlinks 
created


~$ python --version
Python 2.7.10

2. Then I tried to update /pip/, uninstall and re-install /pyignite/

~$ pip install -U pip
-bash: /Library/Frameworks/Python.framework/Versions/3.7/bin/pip: No 
such file or directory

~$ pip3 install -U pip
Requirement already up-to-date: pip in 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages 
(18.0)


~$ pip3 uninstall pyignite
Uninstalling pyignite-0.3.0:
   Would remove:
 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyignite-0.3.0.dist-info/*
 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pyignite/*

Proceed (y/n)? y
   Successfully uninstalled pyignite-0.3.0

~$ pip3 install pyignite
Collecting pyignite
   Using cached 
https://files.pythonhosted.org/packages/f1/0f/5669cd63fb37fa2025110f61598450567d04a72c8cf5b76bb0ca20c21734/pyignite-0.3.0-py3-none-any.whl
Requirement already satisfied: attrs==18.1.0 in 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages 
(from pyignite) (18.1.0)
Requirement already satisfied: typing==3.6.4 in 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages 
(from pyignite) (3.6.4)

Installing collected packages: pyignite
Successfully installed pyignite-0.3.0


How can I fix this?

On Wed, Sep 19, 2018 at 6:43 PM, Dmitry Melnichuk 
mailto:dmitry.melnic...@nobitlost.com>> 
wrote:


Prachi,

This line in your message

> Requirement already satisfied: pyignite in
> ./Downloads/ignite-python/modules/platforms/python (0.3.1)

looks like you already did an installation of pyignite in this
environment before (maybe with "pip install -e ") from 'Downloads' folder, then delete or move downloaded
copy, and than tried to install pyignite again, this time from PyPI
("pip install ").

pip 

Re: Python thin client installation instructions

[Dmitry Melnichuk]

Prachi,

This line in your message

> Requirement already satisfied: pyignite in
> ./Downloads/ignite-python/modules/platforms/python (0.3.1)

looks like you already did an installation of pyignite in this 
environment before (maybe with "pip install -e ") 
from 'Downloads' folder, then delete or move downloaded copy, and than 
tried to install pyignite again, this time from PyPI ("pip install 
").


pip does not work this way. You should either undo the previous install 
("pip uninstall "), use "--update" argument ("pip install 
--update "), or even better − use virtualenv to create a 
disposable Python environment for every experiment. I provided a link to 
virtualenv manual in the README file and in docs, in the 'Basics → 
Testing' section.


> Also, the installation instruction you have provided are for unix users.
> What are the installation instructions for Windows users?

I tried to be OS-agnostic in the docs. Please tell me, what part of my 
instructions does not work on Windows.


On 9/20/18 7:32 AM, Prachi Garg wrote:

Hi Dmitry,

I tried to follow the instructions for the Python thin client 
installation [1].


~$ pip install pyignite
Requirement already satisfied: pyignite in 
./Downloads/ignite-python/modules/platforms/python (0.3.1)
Requirement already satisfied: typing==3.6.4 in 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages 
(from pyignite) (3.6.4)
Requirement already satisfied: attrs==18.1.0 in 
/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages 
(from pyignite) (18.1.0)


But when I try to run an example, I get an error.

~/Downloads/ignite-python/modules/platforms/python/examples$ python 
get_and_put.py

Traceback (most recent call last):
   File "get_and_put.py", line 16, in 
     from pyignite import Client
ImportError: No module named pyignite


What else need to be done? I am documenting the instructions on 
readme.io , but I need to be able to run a few 
examples first.


Also, the installation instruction you have provided are for unix users. 
What are the installation instructions for Windows users?


[1] 
https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#installation

Re: Python thin client

[Dmitry Melnichuk]

Igor,

All examples are in 'ignite/modules/platforms/python/examples'.

I put examples in separate Python files mostly to be able to 
automatically confirm their operability. All the lengthy explanations 
and cross-references are in the main documentation:


https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html

To make it easier for end users to navigate in pyignite directories, I 
added a small README file to the 'examples' directory with a short 
description and a link to the docs:


https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/readme.md

If you think this README is lacking something regardless of the main 
pyignite docs, please let me know.


On 9/17/18 8:59 PM, Igor Sapego wrote:

Dmitry,

Sorry, I was not clear enough. What I mean is that Ignite distributed
by both source and binary releases. Binary releases contain platforms
code, which is needed to write your own application in the language,
but does not contain developer stuff, such as tests, documentation
generating scripts, etc.

Of course, it is more common to use package managers when
developing your application, and of course, we are going to support
this approach. But still, we provide a way for a user to get any client
without any package manager.

I like the idea of supplying whl package in a binary release, though it
looks like it's going to take more effort to implement it. But I believe
except for the whl package, we will need to supply examples and
instructions, how user can run them.

What do you think?

Best Regards,
Igor


On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Igor,

I am in doubt here because I am not fully comprehend the meaning of
"binary release". But if it is somehow related to the "distribution"
thing, I would dare to suggest the following options:

1. Copy nothing. Just do

```
$ python setup.py bdist_wheel
$ twine upload dist/*
```

during the build process (or separately) and let PyPI handle the
distribution.

This is the most natural and user-friendly way of distributing Python
packages. End user might only do

```
$ pip install pyignite
```

as it is suggested by my readme file.

2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist'
directory, that should appear there as the "python setup.py bdist_wheel"
command finishes. Actually, it can be combined with the first option.

The wheel can then be installed by the end user:

```
$ pip install pyignite-*.whl
```

3. Supply the following files:

ignite/modules/platforms/python/pyignite/**
ignite/modules/platforms/python/requirements/**
ignite/modules/platforms/python/LICENSE
ignite/modules/platforms/python/README.md
ignite/modules/platforms/python/setup.py

(** stands for "all the files and sub-folders recursively, starting from
this folder".)

It is not uncommon or wrong to distribute Python programs as text
without prior packaging, but, in my personal experience, this is a way
more suitable for one-time scripts or proprietary programs. For example,
most of Python web apps relies on git for deployment, without any
packaging subsystem.

Here's a few words about wheels:

https://wheel.readthedocs.io/

Here's about uploading to PyPI with twine:


https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives

On 9/14/18 9:05 PM, Igor Sapego wrote:

Ok, good.

Now, what is about installation? Which directories/files
need to be copied to ignite's binary release?

Best Regards,
Igor

Re: Python thin client

[Dmitry Melnichuk]

Igor,

I am in doubt here because I am not fully comprehend the meaning of 
"binary release". But if it is somehow related to the "distribution" 
thing, I would dare to suggest the following options:


1. Copy nothing. Just do

```
$ python setup.py bdist_wheel
$ twine upload dist/*
```

during the build process (or separately) and let PyPI handle the 
distribution.


This is the most natural and user-friendly way of distributing Python 
packages. End user might only do


```
$ pip install pyignite
```

as it is suggested by my readme file.

2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist' 
directory, that should appear there as the "python setup.py bdist_wheel" 
command finishes. Actually, it can be combined with the first option.


The wheel can then be installed by the end user:

```
$ pip install pyignite-*.whl
```

3. Supply the following files:

ignite/modules/platforms/python/pyignite/**
ignite/modules/platforms/python/requirements/**
ignite/modules/platforms/python/LICENSE
ignite/modules/platforms/python/README.md
ignite/modules/platforms/python/setup.py

(** stands for "all the files and sub-folders recursively, starting from 
this folder".)


It is not uncommon or wrong to distribute Python programs as text 
without prior packaging, but, in my personal experience, this is a way 
more suitable for one-time scripts or proprietary programs. For example, 
most of Python web apps relies on git for deployment, without any 
packaging subsystem.


Here's a few words about wheels:

https://wheel.readthedocs.io/

Here's about uploading to PyPI with twine:

https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives

On 9/14/18 9:05 PM, Igor Sapego wrote:

Ok, good.

Now, what is about installation? Which directories/files
need to be copied to ignite's binary release?

Best Regards,
Igor

Re: Python thin client

[Dmitry Melnichuk]

Igor,

The commented code (lines 95-96) gives an error if executed. The error 
is stated just below, in lines 98-100. It is explained here:


https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html#create

I found out by trial and error that a cache, created with SQL DDL 
statement (`CREATE TABLE`), can be deleted only with SQL DDL statement 
(`DROP TABLE`), whereas a table, created with a binary protocol 
operation (`OP_CACHE_CREATE_*` or `OP_CACHE_GET_OR_CREATE_*`) can be 
deleted only with binary protocol operation (`OP_CACHE_DESTROY`). I miss 
this fact in Ignite documentation.


In this particular part of the example I try to stress this out with 
regard to the cache created in the beginning of the example. The cache 
behaves like an SQL table, but can not be dropped. You need to destroy 
it instead (line 102).


I would not go into such depths at all, but the examples are designed to 
be runnable in automated environments, and thus must have some cleanup 
code. And since the cleanup here is not absolutely trivial, I thought I 
must explain it.


I also could handle an error instead of commenting the erroneous code, like

```
any_error = None
try:
DROP_QUERY = 'DROP TABLE Student'
client.sql(DROP_QUERY)
except Exception as e:
any_error = e
print(any_error)

# pyignite.exceptions.SQLError: class 
org.apache.ignite.IgniteCheckedException:

# Only cache created with CREATE TABLE may be removed with DROP TABLE
# [cacheName=SQL_PUBLIC_STUDENT]

but it would complicate the cleanup part of the example to the point it 
is lost any exemplariness. So I decided to simply use comments.


On 9/14/18 2:48 AM, Igor Sapego wrote:

Ok, now everything's running.

API looks good to me. I have a single question about example code:
What these comments are for - [1]?

[1] -
https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/create_binary.py#L95

Best Regards,
Igor


On Thu, Sep 13, 2018 at 12:52 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Igor,

Yes, it's my bad, sorry. Just merged the Ignite master with my branch.

On 9/12/18 8:47 PM, Igor Sapego wrote:

Pavel,
Yes, I did. I tried completely clean environment, followed the same
steps and got the same error. Then I removed attr, and out of sudden
everything started working.

Dmitry,
Thanks, now it's more clear:
Handshake error: Unsupported version. Server expects binary protocol
version 1.1.0. Client provides 1.2.0.

Why do you use version 1.2.0, but have outdated server code? I
propose you to merge with or rebase onto Ignite's master branch.

Best Regards,
Igor


On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Igor,

I have just commited an improvment to the HandshakeError message
generation algorithm. I hope it is now easier to understand what expects
what in case of binary protocol version mismatch.

Thank you for pointing this out.

On 9/12/18 2:13 AM, Igor Sapego wrote:

I managed to start tests, and now I'm getting the following message:

pyignite.exceptions.HandshakeError: Handshake error: Unsupported
version. Expected protocol version: 1.1.0.

It would be useful to print "Unexpected version" itself, because I can
not understand what is the issue.

Best Regards,
Igor

Re: Python thin client

[Dmitry Melnichuk]

Igor,

Yes, it's my bad, sorry. Just merged the Ignite master with my branch.

On 9/12/18 8:47 PM, Igor Sapego wrote:

Pavel,
Yes, I did. I tried completely clean environment, followed the same
steps and got the same error. Then I removed attr, and out of sudden
everything started working.

Dmitry,
Thanks, now it's more clear:
Handshake error: Unsupported version. Server expects binary protocol
version 1.1.0. Client provides 1.2.0.

Why do you use version 1.2.0, but have outdated server code? I
propose you to merge with or rebase onto Ignite's master branch.

Best Regards,
Igor


On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Igor,

I have just commited an improvment to the HandshakeError message
generation algorithm. I hope it is now easier to understand what expects
what in case of binary protocol version mismatch.

Thank you for pointing this out.

On 9/12/18 2:13 AM, Igor Sapego wrote:

I managed to start tests, and now I'm getting the following message:

pyignite.exceptions.HandshakeError: Handshake error: Unsupported
version. Expected protocol version: 1.1.0.

It would be useful to print "Unexpected version" itself, because I can
not understand what is the issue.

Best Regards,
Igor

Re: Python thin client

[Dmitry Melnichuk]

Igor,

I have just commited an improvment to the HandshakeError message 
generation algorithm. I hope it is now easier to understand what expects 
what in case of binary protocol version mismatch.


Thank you for pointing this out.

On 9/12/18 2:13 AM, Igor Sapego wrote:

I managed to start tests, and now I'm getting the following message:

pyignite.exceptions.HandshakeError: Handshake error: Unsupported
version. Expected protocol version: 1.1.0.

It would be useful to print "Unexpected version" itself, because I can
not understand what is the issue.

Best Regards,
Igor

Re: Python thin client

[Dmitry Melnichuk]

Igor,

I literally followed the steps you describe, but unfortunately could not 
reproduce the error.


The only clue I got is that your site-packages already have the newer 
version of attrs (18.2.0 against 18.1.0 as of pyignite requirements). 
But this should not be an issue, since pytest-runner in this command 
creates its own test environment, independent of the one it runs into. 
It worked for me when I installed `attrs==18.2.0` locally and run tests 
− everything went smooth.


You can try uninstalling or downgrading your local attrs with

```
$ pip3 uninstall attrs
```

but I don't think it will actually help. Maybe it will raise another 
error, so we could dig deeper into the problem.


Alternatively, you can give me more details to recreate it (OS, Python 
and pip versions). I'm not ruling out a bug in setuptools.


On 9/11/18 2:13 AM, Igor Sapego wrote:

Guys, I've cloned your repository, run pip3 install -e .
then run pip3 install -r requirements/* over all the requirements,
and finally run python ./setup.py pytest, but get the following output:

running pytest
Searching for attrs==18.1.0
Best match: attrs 18.1.0

Using /home/isapego/.local/lib/python3.6/site-packages
running egg_info
writing pyignite.egg-info/PKG-INFO
writing dependency_links to pyignite.egg-info/dependency_links.txt
writing requirements to pyignite.egg-info/requires.txt
writing top-level names to pyignite.egg-info/top_level.txt
reading manifest file 'pyignite.egg-info/SOURCES.txt'
writing manifest file 'pyignite.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
   File "./setup.py", line 98, in 
 'Operating System :: OS Independent',
   File
"/home/isapego/.local/lib/python3.6/site-packages/setuptools/__init__.py",
line 140, in setup
 return distutils.core.setup(**attrs)
   File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
 dist.run_commands()
   File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
 self.run_command(cmd)
   File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
 cmd_obj.run()
   File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py", line 175,
in run
 with self.project_on_sys_path():
   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
 return next(self.gen)
   File
"/home/isapego/.local/lib/python3.6/site-packages/setuptools/command/test.py",
line 166, in project_on_sys_path
 require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
   File
"/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 895, in require
 needed = self.resolve(parse_requirements(requirements))
   File
"/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 786, in resolve
 raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (attrs 18.2.0
(/home/isapego/.local/lib/python3.6/site-packages),
Requirement.parse('attrs==18.1.0'), {'pyignite'})

What am I doing wrong?

Best Regards,
Igor

Re: Can not gather default cluster on localhost

[Dmitry Melnichuk]

Maxim,

Thank you very much, your command did it. I did not had to set the 
`IGNITE_HOME` var though.


Now I must analyze the difference it made to put in into my client's 
documentation.


On 9/8/18 2:27 AM, Maxim Muzafarov wrote:

Dmitry,

Recently, I've faced the same issue. I've tried configs from `example`
Ignite directory and they worked for me.
Try: ./ignite.sh -v example/config/example-ignite.xml

Note. You should have the correct IGNITE_HOME environment variable pointing
at yours Ignite dir.

Is this is what you need?

Re: Can not gather default cluster on localhost

[Dmitry Melnichuk]

Alexey,

Yes, I believe I am using the default config. I do not modify any file 
of those came with Ignite nightly build, and I do not supply my own .xml 
to `bin/ignite.sh` script, so I think I'm clean.


Could you please clarify, which exactly file to edit and how to set 
`TcpDiscoveryVmIpFinder` up with proper value? I am not a Java developer 
and not familiar with Ignite fine tuning.


On 9/7/18 6:54 PM, Alexey Goncharuk wrote:

Hello Dmitriy,

Did you use the default config shipped with Ignite? Can you try modifying
the config to use TcpDiscoveryVmIpFinder providing only 127.0.0.1:47500 in
the list of IP addresses?

чт, 6 сент. 2018 г. в 22:16, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:


Hello igniters!

I am again in need of your mighty helping hand.

Recently I experienced a strange bug. It may be a fresh regression, or
maybe I myself did something wrong. It would be great if someone looked
into this matter.

I have a need of testing some things in my client against a cluster of
multiple Ignite instances. If my understanding is correct, I can create
an Ignite cluster on localhost just by launching multiple
`bin/ignite.sh` in console(s) with no special parameters. I tried doing
so, but each Ignite instance I created was forming a separate cluster,
and no data sharing occurred.

I was told that in some cases IPv6 may prevent default Ignite instances
from gathering into a cluster, at least there was a same or similar case
with Ignite 2.7 on fresh MacOS X. I disabled IPv6 with

```
$ _JAVA_OPTIONS=-Djava.net.preferIPv4Stack=true bin/ignite.sh
```

but the outcome did not change. For me it looks like clustering is
enabled, but instances can not detect each other's presence.

My OS is Arch Linux. This is my java version:

```
$ java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
```

Output of the first console:

```
  >>> +--+
  >>> Ignite ver. 2.7.0.20180830#19700101-sha1:DEV
  >>> +--+
  >>> OS name: Linux 4.18.3-arch1-1-ARCH amd64
  >>> CPU(s): 4
  >>> Heap: 1.0GB
  >>> VM name: 20751@ibmpc
  >>> Local node [ID=FD43A01E-C0D1-4ED3-A653-036982C0205F, order=1,
clientMode=false]
  >>> Local node addresses: [ibmpc/127.0.0.1, ibmpc/192.168.0.222,
/192.168.122.1]
  >>> Local ports: TCP:8080 TCP:10800 TCP:11211 TCP:47100 UDP:47400
TCP:47500

[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager] Topology
snapshot [ver=1, servers=1, clients=0, CPUs=4, offheap=3.1GB, heap=1.0GB]
[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager]   ^-- Node
[id=FD43A01E-C0D1-4ED3-A653-036982C0205F, clusterState=ACTIVE]
[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager] Data Regions
Configured:
[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager]   ^-- default
[initSize=256.0 MiB, maxSize=3.1 GiB, persistenceEnabled=false]

```

Second console:
```
  >>> +--+
  >>> Ignite ver. 2.7.0.20180830#19700101-sha1:DEV
  >>> +--+
  >>> OS name: Linux 4.18.3-arch1-1-ARCH amd64
  >>> CPU(s): 4
  >>> Heap: 1.0GB
  >>> VM name: 20168@ibmpc
  >>> Local node [ID=108D943D-CE66-4A5D-B8DD-F59F199F4E66, order=1,
clientMode=false]
  >>> Local node addresses: [ibmpc/127.0.0.1, ibmpc/192.168.0.222,
/192.168.122.1]
  >>> Local ports: TCP:8081 TCP:10801 TCP:11212 TCP:47101 UDP:47400
TCP:47501

[07-09-2018 03:37:13][INFO ][main][GridDiscoveryManager] Topology
snapshot [ver=1, servers=1, clients=0, CPUs=4, offheap=3.1GB, heap=1.0GB]
[07-09-2018 03:37:13][INFO ][main][GridDiscoveryManager]   ^-- Node
[id=108D943D-CE66-4A5D-B8DD-F59F199F4E66, clusterState=ACTIVE]
```

Third console:
`>>> +--+
  >>> Ignite ver. 2.7.0.20180830#19700101-sha1:DEV
  >>> +--+
  >>> OS name: Linux 4.18.3-arch1-1-ARCH amd64
  >>> CPU(s): 4
  >>> Heap: 1.0GB
  >>> VM name: 20433@ibmpc
  >>> Local node [ID=40D54138-491D-4AF9-BD10-5A3B53E38E77, order=1,
clientMode=false]
  >>> Local node addresses: [ibmpc/127.0.0.1, ibmpc/192.168.0.222,
/192.168.122.1]
  >>> Local ports: TCP:8082 TCP:10802 TCP:11213 TCP:47102 UDP:47400
TCP:47502

[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager] Topology
snapshot [ver=1, servers=1, clients=0, CPUs=4, offheap=3.1GB, heap=1.0GB]
[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager]   ^-- Node
[id=40D54138-491D-4AF9-BD10-5A3B53E38E77, clusterState=ACTIVE]
[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager] Data Regions
Configured:
[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager]   ^-- default
[initSize=256.0 MiB, maxSize=3.1 GiB, persistenceEnabled=false]``

```

Dmitry

Can not gather default cluster on localhost

[Dmitry Melnichuk]

Hello igniters!

I am again in need of your mighty helping hand.

Recently I experienced a strange bug. It may be a fresh regression, or 
maybe I myself did something wrong. It would be great if someone looked 
into this matter.


I have a need of testing some things in my client against a cluster of 
multiple Ignite instances. If my understanding is correct, I can create 
an Ignite cluster on localhost just by launching multiple 
`bin/ignite.sh` in console(s) with no special parameters. I tried doing 
so, but each Ignite instance I created was forming a separate cluster, 
and no data sharing occurred.


I was told that in some cases IPv6 may prevent default Ignite instances 
from gathering into a cluster, at least there was a same or similar case 
with Ignite 2.7 on fresh MacOS X. I disabled IPv6 with


```
$ _JAVA_OPTIONS=-Djava.net.preferIPv4Stack=true bin/ignite.sh
```

but the outcome did not change. For me it looks like clustering is 
enabled, but instances can not detect each other's presence.


My OS is Arch Linux. This is my java version:

```
$ java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
```

Output of the first console:

```
>>> +--+
>>> Ignite ver. 2.7.0.20180830#19700101-sha1:DEV
>>> +--+
>>> OS name: Linux 4.18.3-arch1-1-ARCH amd64
>>> CPU(s): 4
>>> Heap: 1.0GB
>>> VM name: 20751@ibmpc
>>> Local node [ID=FD43A01E-C0D1-4ED3-A653-036982C0205F, order=1, 
clientMode=false]
>>> Local node addresses: [ibmpc/127.0.0.1, ibmpc/192.168.0.222, 
/192.168.122.1]
>>> Local ports: TCP:8080 TCP:10800 TCP:11211 TCP:47100 UDP:47400 
TCP:47500


[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager] Topology 
snapshot [ver=1, servers=1, clients=0, CPUs=4, offheap=3.1GB, heap=1.0GB]
[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager]   ^-- Node 
[id=FD43A01E-C0D1-4ED3-A653-036982C0205F, clusterState=ACTIVE]
[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager] Data Regions 
Configured:
[07-09-2018 03:39:41][INFO ][main][GridDiscoveryManager]   ^-- default 
[initSize=256.0 MiB, maxSize=3.1 GiB, persistenceEnabled=false]


```

Second console:
```
>>> +--+
>>> Ignite ver. 2.7.0.20180830#19700101-sha1:DEV
>>> +--+
>>> OS name: Linux 4.18.3-arch1-1-ARCH amd64
>>> CPU(s): 4
>>> Heap: 1.0GB
>>> VM name: 20168@ibmpc
>>> Local node [ID=108D943D-CE66-4A5D-B8DD-F59F199F4E66, order=1, 
clientMode=false]
>>> Local node addresses: [ibmpc/127.0.0.1, ibmpc/192.168.0.222, 
/192.168.122.1]
>>> Local ports: TCP:8081 TCP:10801 TCP:11212 TCP:47101 UDP:47400 
TCP:47501


[07-09-2018 03:37:13][INFO ][main][GridDiscoveryManager] Topology 
snapshot [ver=1, servers=1, clients=0, CPUs=4, offheap=3.1GB, heap=1.0GB]
[07-09-2018 03:37:13][INFO ][main][GridDiscoveryManager]   ^-- Node 
[id=108D943D-CE66-4A5D-B8DD-F59F199F4E66, clusterState=ACTIVE]

```

Third console:
`>>> +--+
>>> Ignite ver. 2.7.0.20180830#19700101-sha1:DEV
>>> +--+
>>> OS name: Linux 4.18.3-arch1-1-ARCH amd64
>>> CPU(s): 4
>>> Heap: 1.0GB
>>> VM name: 20433@ibmpc
>>> Local node [ID=40D54138-491D-4AF9-BD10-5A3B53E38E77, order=1, 
clientMode=false]
>>> Local node addresses: [ibmpc/127.0.0.1, ibmpc/192.168.0.222, 
/192.168.122.1]
>>> Local ports: TCP:8082 TCP:10802 TCP:11213 TCP:47102 UDP:47400 
TCP:47502


[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager] Topology 
snapshot [ver=1, servers=1, clients=0, CPUs=4, offheap=3.1GB, heap=1.0GB]
[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager]   ^-- Node 
[id=40D54138-491D-4AF9-BD10-5A3B53E38E77, clusterState=ACTIVE]
[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager] Data Regions 
Configured:
[07-09-2018 03:38:23][INFO ][main][GridDiscoveryManager]   ^-- default 
[initSize=256.0 MiB, maxSize=3.1 GiB, persistenceEnabled=false]``


```

Dmitry

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hello, Prachi!

The section is a hand-written .rst-file, which contains links to some 
content, generated from code with `sphinx-apidoc`.


All autogenerated content resides in `source` folder. For example, this 
is an autogenerated page:


https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/source/pyignite.client.html

And this is written by hand:

https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html

On 9/6/18 9:39 AM, Prachi Garg wrote:

Hi Dmitry M,

Is the API Specification [1] section of the documentation auto-generated
from the code?

[1]
https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/modules.html

Re: Python thin client

[Dmitry Melnichuk]

Dmitriy,

It would be quite messy to implement with Python modular system.

First of all, Python 2 and Python 3 are different languages with a small 
common subset of syntax rules. That's why what we see in a stack trace 
is a syntax error, and not a “missing feature” error.


Second, there is no single entry point in Python code. User is allowed 
to import any name, from any module, in any order. In fact the module is 
run when it first discovered by CPython during any `import` operation, 
and that is how the imported entities are created and initialized: by 
the chain of imports.


It means for us, that implementing even a simple compatibility message 
in Python 2 requires any module in our program (or at least all the 
modules, that represent the public API of our library) to consist 
entirely of a valid Python 2 code.


It can be achieved by writing stubs with a version check, putting said 
stubs in place of real modules, and proxying all the calls through the 
conditional imports. It would take a small effort once, but make code 
less readable and harder to maintain for the rest of its life cycle. 
Should we, for example, provide two testing environments − for the main 
program and for Python 2-compatible stubs?


As far as I know, no Python developer is making such efforts nowadays. 
There are some projects with a long history, that achieve 
2/3-compatibility through the use of restricted syntax and external 
packages like `six`, or simply support two separate versions. Most of 
the new projects are creating with the latest Python 3, pip and 
virtualenv in mind.


I took the idea of my `setup.py` solution from the Django project, which 
is dropped Python 2 support not long ago. Its setup relies on 
`setuptools` 9+ option, or otherwise displays a simple banner; the 
banner is likely to be buried under the further cryptic output of old 
setuptools, but it is better than nothing.


On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote:

Dmitriy,

setuptools sounds like an installation step. Does it make sense to add a
check during startup of a client?

D.

On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Nikolay,

There is indeed a feature in `setuptools` I just learned about, which
would help in this case (and I believe the case you demonstrated can be
quite typical, thank you for pointing it out). It gives user a clever
message in the end of a stack trace like


UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
running Python is 2.7.15


This feature is not 100% bullet-proof, and it will not help users who will
try to install my client other way than with `pip` or `setup.py`.
It will also be less helpful with old versions of `pip` (before 9.0).
However, it should cover most situations.

On 9/4/18 7:15 PM, Nikolay Izhikov wrote:


Hello, Dmitry.

I understand that for experienced Python developer it obvious from
stack trace I send.

But can we check python version on startup? And print big fat error
message "You are using wrong python version".





 From my experience, there are some tickets in Ignite that should be
implemented in various thin clients. It a very trivial changes, but
lack of testability makes this task harder then steel.

I think a .Net DEVNOTES are very good example.

Please, be gentle with your fellow contributors and make DEVNOTES as
clear as possible.

Re: Python thin client

[Dmitry Melnichuk]

Nikolay,

There is indeed a feature in `setuptools` I just learned about, which 
would help in this case (and I believe the case you demonstrated can be 
quite typical, thank you for pointing it out). It gives user a clever 
message in the end of a stack trace like


> UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
> running Python is 2.7.15

This feature is not 100% bullet-proof, and it will not help users who 
will try to install my client other way than with `pip` or `setup.py`.

It will also be less helpful with old versions of `pip` (before 9.0).
However, it should cover most situations.

On 9/4/18 7:15 PM, Nikolay Izhikov wrote:

Hello, Dmitry.

I understand that for experienced Python developer it obvious from
stack trace I send.

But can we check python version on startup? And print big fat error
message "You are using wrong python version".

>

From my experience, there are some tickets in Ignite that should be
implemented in various thin clients. It a very trivial changes, but
lack of testability makes this task harder then steel.

I think a .Net DEVNOTES are very good example.

Please, be gentle with your fellow contributors and make DEVNOTES as
clear as possible.

Re: Python thin client

[Dmitry Melnichuk]

Hello, Nikolay!

Thank you for your interest in Python thin client.

The traceback suggests that you using Python 2.7. Unfortunately, my 
client supports only Python versions 3.4 and later.


Since you are using the latest Ubuntu OS, chances are you already have a 
recent Python 3 installed. Try `pip3`.


On 9/3/18 4:25 PM, Nikolay Izhikov wrote:

Hello, Dmitry.

I tried to build your lib locally and it failed.
Error message and pip version are below.

Seems, we have to fix developer instructions.
Do we need specific version of pip or something?
I tried to use default versions.
I using Ubuntu Linux.

dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v -e .
Obtaining file:///home/dragon/src/ignite/modules/platforms/python
   Running setup.py 
(path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info for 
package from file:///home/dragon/src/ignite/modules/platforms/python
 Running command python setup.py egg_info
 Traceback (most recent call last):
   File "", line 1, in 
   File "/home/dragon/src/ignite/modules/platforms/python/setup.py", line 20
 def is_a_requirement(line: str) -> bool:
  ^
 SyntaxError: invalid syntax
Cleaning up...
Command "python setup.py egg_info" failed with error code 1 in 
/home/dragon/src/ignite/modules/platforms/python/
Exception information:
Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
 status = self.run(options, args)
   File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, 
in run
 wb.build(autobuilding=True)
   File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
 self.requirement_set.prepare_files(self.finder)
   File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in 
prepare_files
 ignore_dependencies=self.ignore_dependencies))
   File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in 
_prepare_file
 abstract_dist.prep_for_dist()
   File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in 
prep_for_dist
 self.req_to_install.run_egg_info()
   File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 439, in 
run_egg_info
 command_desc='python setup.py egg_info')
   File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, in 
call_subprocess
 % (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 
in /home/dragon/src/ignite/modules/platforms/python/
dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache policy 
python-pip
python-pip:
   Установлен: 9.0.1-2.3~ubuntu1
   Кандидат:   9.0.1-2.3~ubuntu1
   Таблица версий:
  *** 9.0.1-2.3~ubuntu1 500
 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 
Packages
 500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe i386 
Packages
 100 /var/lib/dpkg/status
  9.0.1-2 500
 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages
 500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 Packages
dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a
Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 
x86_64 x86_64 x86_64 GNU/Linux

Python thin client

[Dmitry Melnichuk]

Hello, Igniters!

Please review my work on Ignite thin client library written in Python 3.

Pull request:
https://github.com/apache/ignite/pull/4278

Jira issue with initial proposal:
https://issues.apache.org/jira/browse/IGNITE-7782

The documentation is temporarily available at:
https://apache-ignite-binary-protocol-client.readthedocs.io/

It covers installation, requirements, API specification, coding 
examples, instructions on how to run tests and build the documentation 
itself.


Dmitry

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Dmitriy, Igor, Ilya, Sergey!

Thank you for sharing your ideas, concerns and criticism with me. I do 
appreciate it.


I already made some changes in my API, influenced by your feedback. I 
also plan to add a certain set of features, that will make my UX closer 
to what you can see from other Ignite clients.


I stopped using `hashcode` in my examples. Integer cache IDs and cache 
names now can be used interchangeably, with primary focus on cache names.


I will add a Cache class as a primary interface for cache operations, so 
that earlier examples:


```
conn = Connection()
conn.connect('127.0.0.1', 10800)

cache_create(conn, 'my cache')

cache_put(conn, 'my cache', 'my key', 42)
result = my_cache.get('my key')

cache_destroy(conn, 'my cache')
conn.close()
```

could be reiterated as:

```
conn = Connection()
conn.connect('127.0.0.1', 10800)

my_cache = conn.create_cache('my cache')

my_cache.put('my key', 42)
result = my_cache.get('my key')

my_cache.destroy('my cache')
conn.close()
```

I will also make `Connection.connect()` accept any iterable (including 
simple list) as a connection parameter. I will provide user with some 
basic connection generators instead of what is done in my current 
connection failover example.


On 07/27/2018 07:41 AM, Dmitriy Setrakyan wrote:

Dmitriy,

I would stop using the word "hashcode" in this context. Hash code has a
special meaning in Ignite and is used to determine key-to-node affinity. I
agree that passing "cache_name" is the best option. I have no idea when
"cache_name" is not going to be known and do not think we need to support
this case at all. My suggestion is to drop the cache_id use case altogether.

Also I am really surprised that we do not have a cache abstraction in
python and need to pass cache name and connection into every method. To be
honest, this smells really bad that such a popular modern language like
Python forces us to have such a clumsy API. Can you please take a look at
the Redis python clients and see if there is a better way to support this?

https://redis.io/clients#python

D.

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hi, Ilya!

I considered this option. Indeed, the code would look cleaner if only 
one kind of identifier (preferably the human-readable name) was used. 
But there can be a hypothetical situation, when the user is left with 
hash code only. (For example, obtained from some other API.) It would be 
sad to have an identifier and not be able to use it.


Now I really think about using hash codes and names interchangeably, so both

```
cache_put(conn, 'my-cache', value=1, key='a')
```

and


```
cache_put(conn, my_hash_code, value=1, key='a')
```

will be allowed.

This will be a minor complication on my side, and quite reasonable one.

On 07/26/2018 10:44 PM, Ilya Kasnacheev wrote:

Hello!

Why not use cache name as string here, instead of cache_id()?

cache_put(conn, 'my-cache', value=1, key='a')

Regards,

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hi Prachi!

At the moment I already have my documents (temporarily) published at 
RTD. This is how they look like at a whole:


https://apache-ignite-binary-protocol-client.readthedocs.io/

I already have a separate section on examples:

https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html

My build process is also documented here

https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/readme.html#documentation

and there

https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/README.md

This instructions works for me, and I have at least one report of 
successful documentation build from elsewhere. And RTD is using 
basically the same way to put my docs online.


My way of document building is pretty common for Python package 
developers, but if it needs some modifications to fit into Ignite 
process, please let me know.


All the document sources (both autodoc'ed and hand-crafted) is available at

https://github.com/nobitlost/ignite/tree/ignite-7782/modules/platforms/python/docs

I will be glad to answer any questions.

On 07/26/2018 06:25 AM, Prachi Garg wrote:

Hi Dmitry M,

I am resposible for managing the Ignite documentation. At some point we
will merge the python documentation on github into the main Ignite
documentation. Currently, I am trying to restructure our thin client
documentation in a way that it (thin client documentation) is consistent
for all supported languages - Java, Node.js, Python etc.

I looked at the python document on github. Under the :mod:`~pyignite.api`
section,  I see all the components - cache config, key value, sql, binary
types - but there are no code snippets. Is it possible for you describe
these components with code examples?

See for example -
https://apacheignite.readme.io/docs/java-thin-client-api#section-sql-queries
where the SQL Queries section explains, with example, how the thin client
SQL API can be used.

Similarly, please see -
https://apacheignite.readme.io/docs/java-thin-client-security
https://apacheignite.readme.io/docs/java-thin-client-high-availability
https://apacheignite.readme.io/docs/java-thin-client-api

Thanks,
-Prachi

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Either

```
conn = Connection('example.com', 10800)
cache_put(conn, cache_id('my-cache'), 'a', 1)
```

or

```
conn = Connection('example.com', 10800)
my_cache_id = cache_id('my-cache')
cache_put(conn, my_cache_id, 'a', 1)
```

It is also possible to give parameters names, if you like to.

```
conn = Connection('example.com', 10800)
cache_put(conn, cache_id('my-cache'), key='a', value=1)
```

This should also work, but not recommended:

```
conn = Connection('example.com', 10800)
cache_put(conn, cache_id('my-cache'), value=1, key='a')
```

All variants can coexist in one user program.

On 07/26/2018 05:46 AM, Dmitriy Setrakyan wrote:

I am still confused. Let's work through an example. Suppose I have a cache
named "my_cache" and I want to put an entry with key "a" and value "1".

In Java, this code will look like this:



*IgniteCache<...> myCache = ignite.cache("my-cache");myCache.put("a", 1);*



How will the same code look in Python?

D.

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Igor,

That is a very good point. It just did not cross my mind during the 
implementation of this function, that the cache identifier can be 
abstract. I will fix that.


On 07/26/2018 01:46 AM, Igor Sapego wrote:

Well, at least name should be changed, IMO, as the API function name
should say what we do, and not how we do it. For example, cache_id()
looks better to me than hashcode().

Best Regards,
Igor

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Dmitriy,

The short answer is: Python client uses hash code (or any cache 
identifier) instead of Java Cache object, since such an abstraction is 
not necessary in Python.


As for why (IMO) Cache object may be needed in Java, but not in Python − 
the main reason is the difference in typing systems of both languages.


On 07/25/2018 06:52 PM, Dmitriy Setrakyan wrote:

Dmitriy,

To be honest, I got a bit lost in such a big email. Can you tell us briefly
- why do we not need the hash code on API in Java and we do need it in
Python?

D.

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Igor,

Thank you for your elaborated response. I think the examples you gave me 
are valid for statically-typed languages only. At least not for Python.


If we add an extra parameter to key-value operation, we just add it to a 
given API function as a named argument (“keyword argument” or “kwarg”) 
with a sensible default value, so all the dependent code will run as it 
did before the change, implicitly using the new default value. The newly 
written code, in turn, will be able to take an advantage of the new 
parameter by setting it explicitly. This behavior may be impossible in 
Java or C++ without some additional abstraction layer for isolating 
caller-supplied parameters from actually used ones. But for any 
correctly written Python code such isolation level is only superfluous 
and misleading.


The same thing is with the second example. If we stop using a string or 
integer values as a cache identifiers and replace them with, say, floats 
or objects, still no dependent code modification is required, taken that 
the new cache identifiers remain unique and hashable in Pythonic sense 
(i. e. it should be easy to test them for non-equality), that is only 
natural for an identifier. All the existing code will continue to work 
without hassle.


The above said is yet more relevant to the end user code, since my API 
atm is mostly single-levelled and has very few internal codependencies 
to maintain.


On 07/25/2018 08:00 PM, Igor Sapego wrote:

Dmitriy,

To me it seems that procedural approach makes it harder to maintaine
code and add new features. For example, imagine that we are adding
new feature for a thin client, and now we need to pass one more cache
parameter when doing any key-value operation, such as put. In your
approach, you will need to rewrite all the Cache API, adding new
argument to all the functions.

To give you another example, it is possible, that in future we will change
the way cache is identified, so the "hash" argument will become
deprecated in your approach.

To put it simple, procedural approach lacks encapsulation, which is
extremely important when you write library code and want to maintain
backward compatibility. It removes the separation between public API,
which we should not change from version to version, and private code,
which we are free to change whenever and however we want.

Best Regards,
Igor

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hello, Dmitriy, Igor!

Dmitriy, as you have noted, most cache API functions are parameterized 
with the hash codes, and the function for creating the hash codes is 
documented. I personally see no harm in using hash codes as it is 
defined by low-level client operations. But it may be better to enforce 
the use of cache names throughout the Python library, or use names and 
hash codes interchangeably where possible.


Igor, you are right, due to the procedural nature of my client library, 
there is no cache class in it. Caches in Ignite, from Python 
perspective, are one-off, unique, immutable objects. The client code 
will not benefit much from associating any additional context with them. 
On the contrary, handling cache objects in its string or integer 
representation may sometimes be advantageous.


I guess, the implied question here is: why I chose a procedural approach 
first of all? Well, I am glad to answer it too, although the answer will 
be more verbose.


Python is often referred as a multi-paradigm programming language, but 
at heart it is a procedural scripting language. It has no obligatory 
object orientation that could influence Java or C# clients' architecture.


It is also worth noting, that Python has such an elaborate and versatile 
built-in types, so its classes (`object` descendants) is never meant to 
be used as plain data containers. That is unlike JavaScript, where 
`object` is a legitimate default complex data type. Any combination of 
list/set/dict/defaultdict/OrderedDict is always a better, more Pythonic 
choice; getter/setter semantics is considered an anti-pattern.


Considering the above, the client API architecture could be chosen 
depending on the application field. Aiming for the most versatility, 
however, I decided to implement lower-level procedural API.


“Lower-level” in this case does not mean extra efforts for the end user. 
Feel free to review the rest of the examples and make sure that those 
code snippets actually take less and do more, comparing to the 
equivalent code using other client APIs.


If that is still not satisfying, any kind of object-oriented wrapper can 
be built on top of the procedural implementation with little time and 
effort. In fact, before taking the present course of actions, I 
considered the following implementations of other services' clients:


- key-value libraries, like redis-py, that have their methods 
incapsulated in connection class,


- RDBMS libraries like psycopg2 with their cursor classes − it could be 
an architectural choice for SQL and scan queries-oriented client,


- boto3 (Amazon S3 client) have its methods incapsulated in bucket 
objects, somewhat analogous to Ignite caches, but notably different: a) 
S3 supports only one data type: file, b) bucket can be reconfigured on 
the fly, unlike Ignite cache,


- there could be other architectural choices too.

I will be glad to answer your further questions. If you have suggestions 
about higher-level abstractions and their use cases, please let me know.


Dmitry

On 07/24/2018 10:43 PM, Dmitriy Setrakyan wrote:> Yes, looks strange? 
Why is the hash code part of API?

>
> On Tue, Jul 24, 2018, 13:38 Igor Sapego  wrote:
>
>> Ok, I've quickly looked through API and I have a questions. Here
>> is the code from the example:
>>
>> cache_name = 'my cache'
>> hash_code = hashcode(cache_name)
>> cache_create(conn, cache_name)
>> result = cache_put(conn, hash_code, 'my key', 42)
>>
>> I'm not familiar with python, so here is the question. Why there is
>> no "cache" class, and functions with hash code are used?
>>
>> Best Regards,
>> Igor

Application of various Complex object types

[Dmitry Melnichuk]

Hello fellow Ignite developers!

As you may already know, I am building Ignite thin client API library in 
Python. I have reached the point where all applicable operations are 
implemented. All data types are also fully covered, except for some 
variants of Complex object.


I have been already told that Ignite stores its SQL tables using a 
Complex object data type. It lets me write parser and builder for that 
exact variant of Complex object used for an SQL row:


- compact footer approach, all fields are named.

Now I wonder if I can find any other real-life usage example of Complex 
objects


- with no schema,
- with full schema in footer,
- with raw data.

Your advice is greatly appreciated.

Ideally I should be able to automate the creation of various kinds of 
Complex objects to build my unit tests upon it.


Dmitry

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hi Igor,

I totally agree with the point that auto-generated things does not 
belong to the source tree.


I already made short instructions on how to generate HTML documents with 
Sphinx in README file. The instructions assume that the user is able to 
use the common tools (Python, pip, virtualenv) and have them installed. 
From the beginning of the development we found it inconvenient for 
those who wish to just read the documents and not fire up the whole 
development process. That's why I included the prebuilt documents in the 
repository.


Note also, that Python programs, unlike Java ones, do not have the 
additional build step and always distributed as text. So each user will 
have to build the documents from the source to read it.


To leverage this condition, most of Python package developers use 
readthedocs.org service (freemium, engine is under MIT license) to 
publish their documentation online. Unfortunately, the service can not 
be used by our project because of the directory structure. RTD treats 
the git root as a home to `docs` directory to build Sphinx docs out of 
it. In our case it would build anything but the documents we need from 
upstream git source. This situation can be fixed by developing Python 
thin client in a separate git repository and including it in upstream as 
a git sub-module, but I see this solution is too radical.


Anyway, I am removing the prebuilt docs from now on, just as you 
suggested. Hope that potential usability problems might be addressed in 
the future.


On 07/02/2018 09:41 PM, Igor Sapego wrote:

By the way, guys,

I can see generated docs under source control. We do not store them
this way in Ignite. Instead, we have separate step during release, where
we generate them and include in binary distribution.

So you should remove documents from Git and provide us with instructions
on how to generate docs during release.

Best Regards,
Igor

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Dmitry,

I have mentioned Jira task in my reply earlier today. Sorry, but the 
task description was a bit messy and less than informative. I just 
updated it. Please have a look.


https://issues.apache.org/jira/browse/IGNITE-7782

Please let me know if this information is supposed to be somewhere in 
Wiki. I'll post it right away.


On 06/29/2018 07:53 AM, Dmitriy Setrakyan wrote:

Is there an IEP for it where the community can read about the feature set
that will be supported by the Python client?

D. 

Re: Thin Client lib: Python

[Dmitry Melnichuk]

Hi, Dmitry!

I guess there will be no IEP specifically for the client, because it has 
nothing to do with Ignite enhancing, as what IEP stands for.


From Ignite side, the purpose and potential capabilities of the thin 
client, aka binary API client, is best described by this document:


https://apacheignite.readme.io/docs/binary-client-protocol

From our side, there is some basic info about our project in the 
corresponding Jira task:


https://issues.apache.org/jira/browse/IGNITE-7782

This info is yet to be extended and adjusted.

For more in-depth look at our current work in progress please consult 
our Sphinx documentation. It is contained in our repo in already 
prebuilt form, and will be updated frequently:


https://github.com/nobitlost/ignite/tree/ignite-7782/modules/platforms/python/docs/generated

Regarding the feature set, we strive for the most complete 
implementation of the protocol. However, some of its features (type IDs, 
filter objects, binary type registering, and may be more) are specific 
for Java typing system and thus not relevant for Python language, and 
will unlikely be implemented.


Regards,

On 06/29/2018 07:53 AM, Dmitriy Setrakyan wrote:

Is there an IEP for it where the community can read about the feature set
that will be supported by the Python client?

D.

Thin Client lib: Python

[Dmitry Melnichuk]

Hello Ignite developers!

I would like to bring to your attention, that the new Apache Ignite 
binary protocol API client library, written in Python 3, is out for your 
review and evaluation.


This client is in an early stage of development and lacks some features, 
but I am going to further develop and improve it in the near future with 
your comments, suggestions, and feedback.


The following is a PR message.

 Forwarded Message 
Subject: [GitHub] ignite pull request #4278: IGNITE-7782 Thin Client 
lib: Python (initial impl...

Date: Thu, 28 Jun 2018 17:31:27 + (UTC)
From: dmelnichuk 
Reply-To: dev@ignite.apache.org, dev@ignite.apache.org
To: dev@ignite.apache.org

GitHub user dmelnichuk opened a pull request:

https://github.com/apache/ignite/pull/4278

IGNITE-7782 Thin Client lib: Python (initial implementation)

Please review this preliminary version of Ignite binary API client 
library, written in Python 3.

What is done:
- all API operations implemented,
- code is mostly tested (pytest, ~90% covered),
- setuptools integration, 100% PyPI-ready,
- documentation provided (Sphinx/autodoc).
What is **not** yet done:
- complex object datatype support,
- authentication and authorization support,
- throughly tested SQL operations,
- improved data checking and error reporting,
- …
Tested with Python 3.4 and 3.6, Ignite 2.5.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nobitlost/ignite ignite-7782

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4278.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4278

commit 4b444dff62c04b97bc2cdb0676ad52880374
Author: ekaterina-nbl 
Date:   2018-03-20T21:12:21Z

initial implementation

commit d1e8014c80f698028d23e09cafd7ea190ac3e929
Author: ekaterina-nbl 
Date:   2018-03-20T21:32:52Z

initial implementation

commit f79229e233ffa7bff1e7c22f04749924af6d712a
Author: ekaterina-nbl 
Date:   2018-03-22T09:39:32Z

initial implementation

commit 658d60b58840080b664e02815f4ba6aac76e5804
Author: ekaterina-nbl 
Date:   2018-03-22T16:52:18Z

minor update

commit 617cef12ad72791c7e7e67179e1b0c3f7f3ae6bb
Author: ekaterina-nbl 
Date:   2018-03-25T13:33:27Z

api spec

commit d9729585f5a901977e2a2e40e86a2b715fab79fa
Author: alexey-nbl 
Date:   2018-03-25T21:27:11Z

link to api_spec added

commit ea847eba62e556fa81cbf9838ffe17af60f464e4
Author: ekaterina-nbl 
Date:   2018-03-31T22:07:50Z

error types modified

commit c2ad53fe625cc3a05155eeef318421538830
Author: ekaterina-nbl 
Date:   2018-03-31T23:41:56Z

client states

commit 6d2233864b4d891361d38a7143846570bd6c0ef6
Author: ekaterina-nbl 
Date:   2018-04-01T13:11:27Z

object types improvement

commit 52fbc5f87143da068596141cb59b17b684fd2c1f
Author: ekaterina-nbl 
Date:   2018-04-02T16:59:52Z

complex objects

commit cae5a28e7e0d610434debcc140738e2f48d061cf
Author: ekaterina-nbl 
Date:   2018-04-02T20:13:00Z

object types improvement

commit 3165405d4eae09519dc9b5f40f162eb74a9b3c5d
Author: ekaterina-nbl 
Date:   2018-04-02T21:21:26Z

client states

commit fdbb8f86b32fe4c038d38620a80921be3038f31f
Author: alexey-nbl 
Date:   2018-04-03T08:26:04Z

Ignite client states described

commit 04b946885db0ea2f6fe56a75e28302641dad5f61
Author: alexey-nbl 
Date:   2018-04-03T09:35:49Z

minor update

commit f4aaf1c3f23c82ba7974b4c571a9984748e5e82e
Author: alexey-nbl 
Date:   2018-04-03T12:47:54Z

Update ObjectType.js

commit e4c8279f4e83b3ed13383420ab3d1417b090a3fa
Author: ekaterina-nbl 
Date:   2018-04-03T13:46:19Z

minor updates + api spec

commit ed2e4ee830ca40a28dc31958665f52fab6a0bcdd
Author: alexey-nbl 
Date:   2018-04-04T14:34:52Z

Update ObjectType.js

commit 7e1666eb5df82b622a028c0ea949fa21c79e66c2
Author: alexey-nbl 
Date:   2018-04-04T15:14:00Z

Update ObjectType.js

commit 7b0270d65b597dc1b0d164e78f07c3a21c37ca67
Author: ekaterina-nbl 
Date:   2018-04-08T17:16:43Z

sql queries, key value ops

commit fe90f53fd08f77add17fbf06d014f9a9b0a11c65
Author: alexey-nbl 
Date:   2018-04-08T18:31:47Z

Update IgniteClient.js

commit 04137bf5ec3b7077e194edd0100a01bb43f7933a
Author: alexey-nbl 
Date:   2018-04-08T18:37:04Z

Update CacheConfiguration.js

commit b63ad5980718da1b0c44fa4ca138c6e85e47aef3
Author: alexey-nbl 
Date:   2018-04-08T19:11:11Z

Update ObjectType.js

commit 756b908c9dc38ae497e4d7d740f836dabed93e48
Author: alexey-nbl 
Date:   2018-04-08T22:41:54Z

Update CacheClient.js

commit e96ffee17298dd25d26a7029738132478271cf92
Author: ekaterina-nbl 
Date:   2018-04-08T23:23:33Z

object array, minor updates

commit c050e671f74232c4efc41f51c2018d08b152cbbc
Author: alexey-nbl 
Date:   2018-04-09T21:04:35Z

Update CacheClient.

Contribution request: Ignite lightweight Python client

[Dmitry Melnichuk]

Hello, my name is Dmitry Melnichuk, my Jira username is "Melnichuk". I 
would like to work on issue IGNITE-7782. Please assign the task to me or 
give me permissons to assign tasks.



Regards,

Dmitry