[issue3353] make built-in tokenizer available via Python C API

[Terry J. Reedy]

Changes by Terry J. Reedy :


--
versions:  -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Meador Inge]

Meador Inge  added the comment:

It would be nice if this same C API was used to implement the 'tokenize' 
module.  Issues like issue2180 will potentially require bug fixes in two places 
:-/

--
nosy: +meadori

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Fredrik Lundh]

New submission from Fredrik Lundh <[EMAIL PROTECTED]>:

CPython provides a Python-level API to the parser, but not to the
tokenizer itself.  Somewhat annoyingly, it does provide a nice C API,
but that's not properly exposed for external modules.  

To fix this, the tokenizer.h file should be moved from the Parser
directory to the Include directory, and the (semi-public) functions that
already available must be flagged with PyAPI_FUNC, as shown below.

The PyAPI_FUNC fix should be non-intrusive enough to go into 2.6 and
3.0; moving stuff around is perhaps better left for a later release
(which could also include a Python binding).

Index: tokenizer.h
===
--- tokenizer.h (revision 514)
+++ tokenizer.h (working copy)
@@ -54,10 +54,10 @@
const char* str;
 };

-extern struct tok_state *PyTokenizer_FromString(const char *);
-extern struct tok_state *PyTokenizer_FromFile(FILE *, char *, char *);
-extern void PyTokenizer_Free(struct tok_state *);
-extern int PyTokenizer_Get(struct tok_state *, char **, char **);
+PyAPI_FUNC(struct tok_state *) PyTokenizer_FromString(const char *);
+PyAPI_FUNC(struct tok_state *) PyTokenizer_FromFile(FILE *, char *,
char *);
+PyAPI_FUNC(void) PyTokenizer_Free(struct tok_state *);
+PyAPI_FUNC(int) PyTokenizer_Get(struct tok_state *, char **, char **);

 #ifdef __cplusplus
 }

--
components: Interpreter Core
messages: 69650
nosy: effbot
severity: normal
status: open
title: make built-in tokenizer available via Python C API
type: feature request
versions: Python 2.6, Python 2.7, Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

IMO the "struct tok_state" should not be part of the API, it contains
too many implementation details. Or maybe as an opaque structure.

--
nosy: +amaury.forgeotdarc

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Fredrik Lundh]

Fredrik Lundh <[EMAIL PROTECTED]> added the comment:

There are a few things in the struct that needs to be public, but that's
nothing that cannot be handled by documentation.  No need to complicate
the API just in case.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Andy]

Andy <[EMAIL PROTECTED]> added the comment:

Sorry for the terribly dumb question about this.

Are you meaning that, at this stage, all that is required is:

 1. the application of the PyAPI_FUNC macro
 2. move the file to the Include directory
 3. update Makefile.pre.in to point to the new location

Just I have read this now 10 times or so and keep thinking more must be
involved :-) [certainly given my embarrassing start to the Python dev
community re:asynchronous thread exceptions :-| ]

I have attached a patch that does this. Though at this time it is
lacking any documentation that will state what parts of "struct
tok_state" are private and public. I will need to trawl the code some
more to do that.

I have executed:

 - ./configure
 - make
 - make test

And all proceed well.

--
keywords: +patch
nosy: +kirkshorts
Added file: http://bugs.python.org/file10961/issue3353.diff

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Fredrik Lundh]

Fredrik Lundh <[EMAIL PROTECTED]> added the comment:

That's should be all that's needed to expose the existing API, as is. 
If you want to verify the build, you can grab the pytoken.c and setup.py
files from this directory, and try building the module.

http://svn.effbot.org/public/stuff/sandbox/pytoken/

Make sure you remove the local copy of "tokenizer.h" that's present in
that directory before you build.  If that module builds, all's well.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Andy]

Andy <[EMAIL PROTECTED]> added the comment:

Did that and it builds fine.

So my test procedure was:

 - checkout clean source
 - apply patch as per guidelines
 - remove the file Psrser/tokenizer.h (*)
 - ./configure
 - make
 - ./python setup.py install

Build platform: Ubuntu , gcc 4.2.3

All works fine.

thanks for the extra test files.

* - one question though. I removed the file using 'svn remove' but the
diff makes it an empty file not removed why is that? (and is it correct?)

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3353] make built-in tokenizer available via Python C API

[Daniel Diniz]

Changes by Daniel Diniz :


--
priority:  -> normal
stage:  -> test needed
versions: +Python 3.2 -Python 2.6, Python 3.0

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com