Re: Problem with an huge dictionary
keobox wrote: On 20 Mag, 12:58, Thomas Lehmann t.lehm...@rtsgroup.net wrote: The question is: Is there a limit on the number of entries a dictionary can have i jython? I wrote a little app where my data is stored in a huge dictionary (11746 entries) generated with a python script. When I try to import the dictionary, jython complains with the following message: 1) You did not say what you have saved (content of your dictionary). - As you can see I received the error when I tried to import jmoco_event_data.py module from the jython interpreter prompt. - The import will cause the module compilation, jython will produce a .class file instead of .pyc. - jmoco_event_data.py contains 4 dictionaries: typesValToName={ 220:'EMS_EVENT_EMS_INTERNAL_BASE', 221:'EMS_EVENT_INTERNAL_TYPE_BASE', 222:'EMS_EVENT_INTERNAL_N_E_P_M_EVENT', 223:'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD', 224:'EMS_EVENT_NE_SPECIFIC_BASE', ... 11746 entries int key: string value } typesNameToVal={ 'EMS_EVENT_EMS_INTERNAL_BASE':220, 'EMS_EVENT_INTERNAL_TYPE_BASE':221, 'EMS_EVENT_INTERNAL_N_E_P_M_EVENT':222, 'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD':223, 'EMS_EVENT_NE_SPECIFIC_BASE':224, ... total 11746 entries string key: int value } causesValToName={ 0:'NOT_APPLICABLE_UNKNOWN', 1:'SOFTWARE_CAUSE_UNKNOWN', 2:'ABSENT_MODULE', 3:'FAULTY_MODULE', ... 483 entries int key: string value } causesNameToVal={ 'NOT_APPLICABLE_UNKNOWN':0, 'SOFTWARE_CAUSE_UNKNOWN':1, 'ABSENT_MODULE':2, 'FAULTY_MODULE':3, ... 483 entries string key: int value } 2) You did not say how you have saved. The dictionaries are in the jmoco_event_data.py module. From the callstack - also I have never used jython - it looks like that there is a try to create a class. It looks like - I may be wrong - that you have saved user objects in your dictionary - have you? Nope, the dictionaries are only int to string mappings and string to int mappings If so you might fail on loading those objects - especially when your program does not have the code for it. The failure happens during module's import, so my question is: Is jython able to handle such big dictionaries? The problem seems to be the size of the code, i. e. the dict literal rather than the size of the dictionary. With a slightly smaller dict I get $ cat make_dict_source.py from itertools import chain, count, islice def main(): import argparse parser = argparse.ArgumentParser() parser.add_argument(n, type=int, default=100) parser.add_argument(py) args = parser.parse_args() prefix = [data = {\n] suffix = [}\n] pairs = ('enum_whatever_{0}': {0},\n.format(i) for i in count()) pairs = islice(pairs, args.n) with open(args.py, w) as outstream: lines = chain(prefix, pairs, suffix) outstream.writelines(lines) if __name__ == __main__: main() $ python make_dict_source.py 4000 tmp.py $ jython tmp.py Traceback (innermost last): (no code object) at line 0 java.lang.ClassFormatError: Invalid method Code length 71912 in class file org/python/pycode/_pyx0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:632) at org.python.core.BytecodeLoader2.loadClassFromBytes(Unknown Source) at org.python.core.BytecodeLoader.makeClass(Unknown Source) at org.python.core.BytecodeLoader.makeCode(Unknown Source) at org.python.core.Py.compile_flags(Unknown Source) at org.python.core.Py.compile_flags(Unknown Source) at org.python.core.__builtin__.execfile_flags(Unknown Source) at org.python.util.PythonInterpreter.execfile(Unknown Source) at org.python.util.jython.main(Unknown Source) java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method Code length 71912 in class file org/python/pycode/_pyx0 $ jython --version Jython 2.2.1 on java1.6.0_0 Jython seems to generate a method that is bigger than the 64K allowed by Java. The simplest workaround is probably to let your module read the dict data from a separate file in e. g. csv format. Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with an huge dictionary
On 20 Mag, 12:58, Thomas Lehmann t.lehm...@rtsgroup.net wrote: The question is: Is there a limit on the number of entries a dictionary can have i jython? I wrote a little app where my data is stored in a huge dictionary (11746 entries) generated with a python script. When I try to import the dictionary, jython complains with the following message: 1) You did not say what you have saved (content of your dictionary). - As you can see I received the error when I tried to import jmoco_event_data.py module from the jython interpreter prompt. - The import will cause the module compilation, jython will produce a .class file instead of .pyc. - jmoco_event_data.py contains 4 dictionaries: typesValToName={ 220:'EMS_EVENT_EMS_INTERNAL_BASE', 221:'EMS_EVENT_INTERNAL_TYPE_BASE', 222:'EMS_EVENT_INTERNAL_N_E_P_M_EVENT', 223:'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD', 224:'EMS_EVENT_NE_SPECIFIC_BASE', ... 11746 entries int key: string value } typesNameToVal={ 'EMS_EVENT_EMS_INTERNAL_BASE':220, 'EMS_EVENT_INTERNAL_TYPE_BASE':221, 'EMS_EVENT_INTERNAL_N_E_P_M_EVENT':222, 'EMS_EVENT_INTERNAL_N_E_ALARM_RCVD':223, 'EMS_EVENT_NE_SPECIFIC_BASE':224, ... total 11746 entries string key: int value } causesValToName={ 0:'NOT_APPLICABLE_UNKNOWN', 1:'SOFTWARE_CAUSE_UNKNOWN', 2:'ABSENT_MODULE', 3:'FAULTY_MODULE', ... 483 entries int key: string value } causesNameToVal={ 'NOT_APPLICABLE_UNKNOWN':0, 'SOFTWARE_CAUSE_UNKNOWN':1, 'ABSENT_MODULE':2, 'FAULTY_MODULE':3, ... 483 entries string key: int value } 2) You did not say how you have saved. The dictionaries are in the jmoco_event_data.py module. From the callstack - also I have never used jython - it looks like that there is a try to create a class. It looks like - I may be wrong - that you have saved user objects in your dictionary - have you? Nope, the dictionaries are only int to string mappings and string to int mappings If so you might fail on loading those objects - especially when your program does not have the code for it. The failure happens during module's import, so my question is: Is jython able to handle such big dictionaries? More details are required... jmoco_event_data.py has a size of roughly 700KB. -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with an huge dictionary
The question is: Is there a limit on the number of entries a dictionary can have i jython? I wrote a little app where my data is stored in a huge dictionary (11746 entries) generated with a python script. When I try to import the dictionary, jython complains with the following message: 1) You did not say what you have saved (content of your dictionary). 2) You did not say how you have saved. From the callstack - also I have never used jython - it looks like that there is a try to create a class. It looks like - I may be wrong - that you have saved user objects in your dictionary - have you? If so you might fail on loading those objects - especially when your program does not have the code for it. More details are required... -- http://mail.python.org/mailman/listinfo/python-list
[jython] Problem with an huge dictionary
Hi, I apologize if this is not the right group for posting question related to jython. Please give me the right directions if this is the case. The question is: Is there a limit on the number of entries a dictionary can have i jython? I wrote a little app where my data is stored in a huge dictionary (11746 entries) generated with a python script. When I try to import the dictionary, jython complains with the following message: Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. C:\Documents and Settings\ceplacan\My Documents\Documents\Source\source \Python\j moco-testjython Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54) [Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.5.0_13 Type help, copyright, credits or license for more information. import jmoco_event_data Traceback (most recent call last): File stdin, line 1, in module java.lang.ClassFormatError: Invalid this class index 3241 in constant pool in cl ass file _1076 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at org.python.core.BytecodeLoader $Loader.loadClassFromBytes(BytecodeLoad er.java:119) at org.python.core.BytecodeLoader.makeClass(BytecodeLoader.java:37) at org.python.core.BytecodeLoader.makeCode(BytecodeLoader.java: 67) at org.python.core.imp.createFromSource(imp.java:288) at org.python.core.imp.loadFromSource(imp.java:514) at org.python.core.imp.find_module(imp.java:411) at org.python.core.imp.import_next(imp.java:635) at org.python.core.imp.import_name(imp.java:746) at org.python.core.imp.importName(imp.java:806) at org.python.core.ImportFunction.__call__(__builtin__.java: 1232) at org.python.core.PyObject.__call__(PyObject.java:367) at org.python.core.__builtin__.__import__(__builtin__.java: 1202) at org.python.core.__builtin__.__import__(__builtin__.java: 1185) at org.python.core.imp.importOne(imp.java:817) at org.python.pycode._pyx1.f$0(stdin:1) at org.python.pycode._pyx1.call_function(stdin) at org.python.core.PyTableCode.call(PyTableCode.java:165) at org.python.core.PyCode.call(PyCode.java:18) at org.python.core.Py.runCode(Py.java:1204) at org.python.core.Py.exec(Py.java:1248) at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:181) at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter .java:89) at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpret er.java:70) at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpret er.java:46) at org.python.util.InteractiveConsole.push(InteractiveConsole.java:110) at org.python.util.InteractiveConsole.interact(InteractiveConsole.java:9 0) at org.python.util.jython.run(jython.java:316) at org.python.util.jython.main(jython.java:129) java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid this class index 3241 in constant pool in class file _1076 What can I do to avoid this? Regards, Cesare -- http://mail.python.org/mailman/listinfo/python-list
Re: [jython] Problem with an huge dictionary
KeoBox, Can i look at the jython file you are executing? the ClassFormatException is not with the size of the dictionary. Jython is failing on an import. thanks, -- Allan Davis Member of NetBeans Dream Team http://wiki.netbeans.org/NetBeansDreamTeam Lead Developer, nbPython http://wiki.netbeans.org/Python http://codesnakes.blogspot.com (my blog) Co-Chair, CajunJUG http://www.cajunjug.org On Mon, May 17, 2010 at 10:09 AM, keobox keo...@gmail.com wrote: Hi, I apologize if this is not the right group for posting question related to jython. Please give me the right directions if this is the case. The question is: Is there a limit on the number of entries a dictionary can have i jython? I wrote a little app where my data is stored in a huge dictionary (11746 entries) generated with a python script. When I try to import the dictionary, jython complains with the following message: Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. C:\Documents and Settings\ceplacan\My Documents\Documents\Source\source \Python\j moco-testjython Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54) [Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.5.0_13 Type help, copyright, credits or license for more information. import jmoco_event_data Traceback (most recent call last): File stdin, line 1, in module java.lang.ClassFormatError: Invalid this class index 3241 in constant pool in cl ass file _1076 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at org.python.core.BytecodeLoader $Loader.loadClassFromBytes(BytecodeLoad er.java:119) at org.python.core.BytecodeLoader.makeClass(BytecodeLoader.java:37) at org.python.core.BytecodeLoader.makeCode(BytecodeLoader.java: 67) at org.python.core.imp.createFromSource(imp.java:288) at org.python.core.imp.loadFromSource(imp.java:514) at org.python.core.imp.find_module(imp.java:411) at org.python.core.imp.import_next(imp.java:635) at org.python.core.imp.import_name(imp.java:746) at org.python.core.imp.importName(imp.java:806) at org.python.core.ImportFunction.__call__(__builtin__.java: 1232) at org.python.core.PyObject.__call__(PyObject.java:367) at org.python.core.__builtin__.__import__(__builtin__.java: 1202) at org.python.core.__builtin__.__import__(__builtin__.java: 1185) at org.python.core.imp.importOne(imp.java:817) at org.python.pycode._pyx1.f$0(stdin:1) at org.python.pycode._pyx1.call_function(stdin) at org.python.core.PyTableCode.call(PyTableCode.java:165) at org.python.core.PyCode.call(PyCode.java:18) at org.python.core.Py.runCode(Py.java:1204) at org.python.core.Py.exec(Py.java:1248) at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:181) at org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter .java:89) at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpret er.java:70) at org.python.util.InteractiveInterpreter.runsource(InteractiveInterpret er.java:46) at org.python.util.InteractiveConsole.push(InteractiveConsole.java:110) at org.python.util.InteractiveConsole.interact(InteractiveConsole.java:9 0) at org.python.util.jython.run(jython.java:316) at org.python.util.jython.main(jython.java:129) java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid this class index 3241 in constant pool in class file _1076 What can I do to avoid this? Regards, Cesare -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list