Re: Pickle segfaults with custom type
Found. I simply forgot:
if (PyType_Ready(&PyFrozenDictIterKey_Type) < 0) {
goto fail;
}
in the frozendict_exec function for the module.
On Fri, 7 Jan 2022 at 20:27, Marco Sulla
wrote:
> I have a custom implementation of dict using a C extension. All works but
> the pickling of views and iter types. Python segfaults if I try to pickle
> them.
>
> For example, I have:
>
>
> static PyTypeObject PyFrozenDictIterKey_Type = {
> PyVarObject_HEAD_INIT(NULL, 0)
> "frozendict.keyiterator", /* tp_name */
> sizeof(dictiterobject), /* tp_basicsize */
> 0, /* tp_itemsize */
> /* methods */
> (destructor)dictiter_dealloc, /* tp_dealloc */
> 0, /* tp_vectorcall_offset */
> 0, /* tp_getattr */
> 0, /* tp_setattr */
> 0, /* tp_as_async */
> 0, /* tp_repr */
> 0, /* tp_as_number */
> 0, /* tp_as_sequence */
> 0, /* tp_as_mapping */
> PyObject_HashNotImplemented,/* tp_hash */
> 0, /* tp_call */
> 0, /* tp_str */
> PyObject_GenericGetAttr,/* tp_getattro */
> 0, /* tp_setattro */
> 0, /* tp_as_buffer */
> Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
> 0, /* tp_doc */
> (traverseproc)dictiter_traverse,/* tp_traverse */
> 0, /* tp_clear */
> 0, /* tp_richcompare */
> 0, /* tp_weaklistoffset */
> PyObject_SelfIter, /* tp_iter */
> (iternextfunc)frozendictiter_iternextkey, /* tp_iternext */
> dictiter_methods, /* tp_methods */
> 0,
> };
>
> This is the backtrace I get with gdb:
>
> #0 PyObject_Hash (v=0x7f043ce15540 ) at
> ../cpython_3_10/Objects/object.c:788
> #1 0x0048611c in PyDict_GetItemWithError (op=0x7f043e1f4900,
> key=key@entry=0x7f043ce15540 )
> at ../cpython_3_10/Objects/dictobject.c:1520
> #2 0x7f043ce227f6 in save (self=self@entry=0x7f043d8507d0,
> obj=obj@entry=0x7f043e1fb0b0, pers_save=pers_save@entry=0)
> at /home/marco/sources/cpython_3_10/Modules/_pickle.c:4381
> #3 0x7f043ce2534d in dump (self=self@entry=0x7f043d8507d0,
> obj=obj@entry=0x7f043e1fb0b0) at
> /home/marco/sources/cpython_3_10/Modules/_pickle.c:4515
> #4 0x7f043ce2567f in _pickle_dumps_impl (module=,
> buffer_callback=, fix_imports=,
> protocol=,
> obj=0x7f043e1fb0b0) at
> /home/marco/sources/cpython_3_10/Modules/_pickle.c:1203
> #5 _pickle_dumps (module=, args=,
> nargs=, kwnames=)
> at /home/marco/sources/cpython_3_10/Modules/clinic/_pickle.c.h:619
>
> and so on. The problematic part is in the second frame. Indeed the code of
> _pickle.c here is:
>
>
> reduce_func = PyDict_GetItemWithError(st->dispatch_table,
> (PyObject *)type);
>
> The problem is that type is NULL. It tries to get the attribute tp_hash
> and it segfaults.
>
> I tried to change the header of the type to:
>
> PyVarObject_HEAD_INIT(&PyType_Type, 0)
>
> This way it works but, as known, it does not compile on Windows.
>
> The strange fact is that pickling the main type works, even if the type is
> NULL, as suggested for a custom type. This is the main type:
>
> PyTypeObject PyFrozenDict_Type = {
> PyVarObject_HEAD_INIT(NULL, 0)
> "frozendict." FROZENDICT_CLASS_NAME,/* tp_name */
> sizeof(PyFrozenDictObject), /* tp_basicsize */
> 0, /* tp_itemsize */
> (destructor)dict_dealloc, /* tp_dealloc */
> 0, /* tp_vectorcall_offset */
> 0, /* tp_getattr */
> 0, /* tp_setattr */
> 0, /* tp_as_async */
> (reprfunc)frozendict_repr, /* tp_repr */
> &frozendict_as_number, /* tp_as_number */
> &dict_as_sequence, /* tp_as_sequence */
> &frozendict_as_mapping, /* tp_as_mapping */
> (hashfunc)frozendict_hash, /* tp_hash */
> 0, /* tp_call */
> 0,
Re: What to write or search on github to get the code for what is written below:
What Dennis Lee is saying I will see to it later.
Right now what mrabarrnett is saying is of use.
Why does the code written below is not giving any output?
xls = ExcelFile('ABC.xlsx')
df = xls.parse(xls.sheet_names[0], index_col=1)
x=df.to_dict()
print (x)
Only the contents of the first column and the column number is required in the
dictionary
--
https://mail.python.org/mailman/listinfo/python-list
Re: What to write or search on github to get the code for what is written below:
On 1/13/22 16:08, Avi Gross via Python-list wrote: > > I am not replying to anything below so I have removed it. > Instead, someone suggested Python which indeed, with lots of work, can open > just about ANY nonsensical file and diddle around and rewrite it, but WHY? well, the topic *was* raised on a PYTHON list... -- https://mail.python.org/mailman/listinfo/python-list
Writing a string with comma in one column of CSV file
Hi, I use the following line to write some information to a CSV file which is comma delimited. f = open(output_file, 'w', newline='') wr = csv.writer(f) ... f.write(str(n) + "," + str(key) + "\n" ) Problem is that key is a string which may contain ',' and this causes the final CSV file to have more than 2 columns, while I want to write the whole key as a single column. I know that wr.writerow([key]) writes the entire key in one column, but I would like to do the same with write(). Any idea to fix that? Regards, Mahmood -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing a string with comma in one column of CSV file
On 16/01/2022 09.56, Mahmood Naderan via Python-list wrote: > Hi, > I use the following line to write some information to a CSV file which is > comma delimited. > > f = open(output_file, 'w', newline='') > wr = csv.writer(f) > ... > f.write(str(n) + "," + str(key) + "\n" ) > > > Problem is that key is a string which may contain ',' and this causes the > final CSV file to have more than 2 columns, while I want to write the whole > key as a single column. > > I know that wr.writerow([key]) writes the entire key in one column, but I > would like to do the same with write(). Any idea to fix that? This is 'CSV' rather than 'Python'. The solution is to "escape" the string. Ref: https://stackoverflow.com/questions/769621/dealing-with-commas-in-a-csv-file -- Regards, =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: What to write or search on github to get the code for what is written below:
Mats, Yes, this is a Python mailing list and I welcome people interested in doing something in Python who need a little help or advice but have some idea of what they are doing and present us with enough info more than "something does not work." Yes, the topic was raised on a Python list but I did not get the impression the person asking necessarily knew a lot about Python or maybe other languages. I may well be wrong. They seem to have been handed something to work on and maybe are searching for some way to do it. You can do what was asked in plenty of languages, some easier and some harder. But some are better set to do things in a browser/server model than others. One impression is they were suggested to use a web platform of sorts and from there went to Python as compatible for the server side. But the overall tone suggests to me that they want to be given a quick solution for a problem they do not really understand, let alone the tools they want to use. And I do not perceive much willingness to take the time to do much learning. My willingness to help and my time to do so, are limited and I said so. Others, feel free. The reality is many here want to be hired for large and complex jobs. If someone had suggested to them to use JavaScript written inside a web page to read the file and do searches, and forgot to tell them that writing the file back onto some server might not be trivial, I wonder if they would have gone in another direction. Again, these are my impressions and why I wondered if there was a GOOD reason they asked about Python. My experience in the past includes seeing posts by people who sent messages to multiple groups I was on for multiple languages and asked how to do something. They just were hoping someone would post complete code and they could then move on without learning anything. Maybe I am a tad jaundiced and prefer honest requests from people willing to listen to what feedback we provide and show some code for us to look at and perhaps help improve. -Original Message- From: Mats Wichmann To: python-list@python.org Sent: Sat, Jan 15, 2022 3:05 pm Subject: Re: What to write or search on github to get the code for what is written below: On 1/13/22 16:08, Avi Gross via Python-list wrote: > > I am not replying to anything below so I have removed it. > Instead, someone suggested Python which indeed, with lots of work, can open > just about ANY nonsensical file and diddle around and rewrite it, but WHY? well, the topic *was* raised on a PYTHON list... -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing a string with comma in one column of CSV file
On Sat, 15 Jan 2022 20:56:22 + (UTC), Mahmood Naderan wrote: > Hi, > I use the following line to write some information to a CSV file which > is comma delimited. > > f = open(output_file, 'w', newline='') > wr = csv.writer(f) > ... > f.write(str(n) + "," + str(key) + "\n" ) > > > Problem is that key is a string which may contain ',' and this causes > the final CSV file to have more than 2 columns, while I want to write > the whole key as a single column. > > I know that wr.writerow([key]) writes the entire key in one column, but > I would like to do the same with write(). Any idea to fix that? > > > Regards, > Mahmood you need to quote the data the easies way to ensure this is to inculde to QUOTE_ALL option when opening the file wr = csv.writer(output, quoting=csv.QUOTE_ALL) -- Chocolate chip. -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing a string with comma in one column of CSV file
Right. I was also able to put all columns in a string and then use writerow(). Thanks. Regards, Mahmood On Saturday, January 15, 2022, 10:33:08 PM GMT+1, alister via Python-list wrote: On Sat, 15 Jan 2022 20:56:22 + (UTC), Mahmood Naderan wrote: > Hi, > I use the following line to write some information to a CSV file which > is comma delimited. > > f = open(output_file, 'w', newline='') > wr = csv.writer(f) > ... > f.write(str(n) + "," + str(key) + "\n" ) > > > Problem is that key is a string which may contain ',' and this causes > the final CSV file to have more than 2 columns, while I want to write > the whole key as a single column. > > I know that wr.writerow([key]) writes the entire key in one column, but > I would like to do the same with write(). Any idea to fix that? > > > Regards, > Mahmood you need to quote the data the easies way to ensure this is to inculde to QUOTE_ALL option when opening the file wr = csv.writer(output, quoting=csv.QUOTE_ALL) -- Chocolate chip. -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing a string with comma in one column of CSV file
Mahmood,
Ask yourself WHY you want to do what you are doing. Are you using the power and
features of the language or trying to do it step by step in the way that
earlier languages often made you do it?
Yes, there are ways to include commas in fields of a CSV file and they can lead
to complications you can completely avoid by NOT using a CSV file. Use
something where commas are not a field separator and something else that is not
likely to be in your data (or when found is removed or perhaps replaced with
something like a space) and can thus be used.
Many files use a TAB or other symbols to delimit data and may be saved with
names like a .TSV file or others. Python can trivially read in such files often
simply by telling it to read something like a CSV but specifying the field
separator is a tab or some other character.
But I have another dumb question. Why are you writing your CSV file by hand and
a line at a time? I mean there is nothing wrong with that but many people have
programs where they make an object like a DataFrame and manipulate that to have
all the data they need and assuming we call the structure df, and they are
using the pandas module, they can write the entire thing out like this:
df.to_csv('new_file.sv', sep='\t', index=False)
The pandas package fairly easily allows you to load in all the data you need
rom an external source, search in what you have, make changes or additions, and
write it out to many kinds of files.
Just a thought. If you like your way, fine, I see another reply suggesting how
to hide the commas but that can be a problem if humans read and edit the
results in the external file and do not follow through.
-Original Message-
From: Mahmood Naderan via Python-list
To: DL Neil via Python-list
Sent: Sat, Jan 15, 2022 3:56 pm
Subject: Writing a string with comma in one column of CSV file
Hi,
I use the following line to write some information to a CSV file which is comma
delimited.
f = open(output_file, 'w', newline='')
wr = csv.writer(f)
...
f.write(str(n) + "," + str(key) + "\n" )
Problem is that key is a string which may contain ',' and this causes the final
CSV file to have more than 2 columns, while I want to write the whole key as a
single column.
I know that wr.writerow([key]) writes the entire key in one column, but I would
like to do the same with write(). Any idea to fix that?
Regards,
Mahmood
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: What to write or search on github to get the code for what is written below:
On 15Jan2022 02:38, NArshad wrote:
>Why does the code written below is not giving any output?
>
>xls = ExcelFile('ABC.xlsx')
>df = xls.parse(xls.sheet_names[0], index_col=1)
>x=df.to_dict()
>print (x)
>
>Only the contents of the first column and the column number is required
>in the dictionary
The code above _must_ produce some output, even if it is very short, or
an error traceback. Is "x" an empty dictionary? Are you getting an
exception? What is in ABC.xlsx, at least in part? What modules are you
importing in order to get the name "ExcelFile"? I'm guessing pandas, but
that is from a web search.
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
