Patch for invenio: dropping tables

[Piotr Praczyk]

Hey !

I have created a small patch for Invenio allowing easier dropping of DB tables.
This patch solves my problems caused by working with different versions of 
Invenio on the same machine and database.
There are 2 main uses:

 1) Someone wants to downgrade Invenio (or in general drop database using 
different version that was used for the creation). In such case tabdrop.sql is 
incompatible with tabcreate.sql used to build the database. 
 2) Someone has tables created by external scripts (or for example some updater 
scripts which create backup tables)

The new option is:
 --drop-all-tables and removes all database tables regardless if they are 
specified in tabdrop.sql
When after user runs inveniocfg with --drop-tables option, some tables remain - 
a warning is generated and names of all remaining tables are listed. (With a 
suggestion that user might want to execute --drop-all-tables)

Could we integrate this mini-branch in master ? It is useful for development 
and maybe for some end-users.
I know that this option is a bit more risky to use... so maybe if it is not 
desired to be publicly visible ( less interesting for end-users than 
developers) - maybe we could make it a hidden undocumented option, yet accepted 
by inveniocfg ? 

The branch is available under the name warning_droptables in my public 
repository or: 
http://invenio-software.org/repo/personal/invenio-ppraczyk/log/?h=warning_droptables


Cheers
Piotr



(ok ;) I have written more e-mail to justify than code ;) ) 

RE: Upgrader script

[Piotr Praczyk]

Just a small amendment to the previous e-mail

I have just realised that in the 2nd question, the logical solution would be in 
every commit consider present scripts as applied...  which basically means that 
inveniocfg --create-tables should always mark all upgrade scripts as applied 
(this seems not to happen)
I misunderstand something or this is a bug ? 


Piotr

From: Piotr Praczyk
Sent: 15 November 2012 21:37
To: project-invenio-devel (Invenio developers mailing-list); 
inspire-...@slac.stanford.edu
Subject: Upgrader script

Hello


I have several questions concerning the Upgrader mechanism introduced to 
Invenio.

First of all, could someone who knows the topic better, review my upgrate 
module ? It can be seen under:

http://invenio-software.org/repo/personal/invenio-ppraczyk/commit/?h=bibdocfile-squashed-ultimate-rebased&id=be09d38416fe5adc0898f40e94fe80118dedd70e



The questions (about what I don't understand after reading the documentation 
 in inveniocfg_upgrader.py) :

1) Modules are supposed to be standalone, not depending too much on imports 
from Invenio "unless it is absolutely necessary".
 - which imports can be considered necessary ? Upgrade scripts cannot be 
ammended in the future (according to the doc), making this work seemingly an 
impossible task.
   For instance (example from this upgrade) if I need to import some 
configuration variables which have got renamed in one of the commits (in this 
example together with moving bibdocfile from websubmit to bibdocfile module).
   If we decide to further rename those varaibles, we have to ammend all 
upgrade scripts, otherwise it will clearly fail.
   Is there a list of safe imports ?

2) How can I mark that a certain commit introduces given changes ?

   Afetr I have installed the commit and created database using invenioconfig 
script, intuitively I would expect everything being up to date and not 
requiring upgrades.
   In reality, the upgrader suggests a series of actions (which cause warnings 
because indeed they seem to be already applied).
   The dependency system seems to be related not to commits but previously run 
scripts.

Is there any mechanism allowing to mark that a given commit gives Invenio 
in the state as after running of a given script ? For example running 
bibdocfile model upgrade should not happen if database was created using newer 
commit from the repository.
   I have not managed to find something like this in the code :(

Piotr

Upgrader script

[Piotr Praczyk]

Hello


I have several questions concerning the Upgrader mechanism introduced to 
Invenio.

First of all, could someone who knows the topic better, review my upgrate 
module ? It can be seen under:

http://invenio-software.org/repo/personal/invenio-ppraczyk/commit/?h=bibdocfile-squashed-ultimate-rebased&id=be09d38416fe5adc0898f40e94fe80118dedd70e



The questions (about what I don't understand after reading the documentation 
 in inveniocfg_upgrader.py) :

1) Modules are supposed to be standalone, not depending too much on imports 
from Invenio "unless it is absolutely necessary".
 - which imports can be considered necessary ? Upgrade scripts cannot be 
ammended in the future (according to the doc), making this work seemingly an 
impossible task.
   For instance (example from this upgrade) if I need to import some 
configuration variables which have got renamed in one of the commits (in this 
example together with moving bibdocfile from websubmit to bibdocfile module).
   If we decide to further rename those varaibles, we have to ammend all 
upgrade scripts, otherwise it will clearly fail.
   Is there a list of safe imports ? 

2) How can I mark that a certain commit introduces given changes ? 

   Afetr I have installed the commit and created database using invenioconfig 
script, intuitively I would expect everything being up to date and not 
requiring upgrades.
   In reality, the upgrader suggests a series of actions (which cause warnings 
because indeed they seem to be already applied).
   The dependency system seems to be related not to commits but previously run 
scripts.
   
Is there any mechanism allowing to mark that a given commit gives Invenio 
in the state as after running of a given script ? For example running 
bibdocfile model upgrade should not happen if database was created using newer 
commit from the repository.
   I have not managed to find something like this in the code :(

Piotr

Broken DELETE-FILE mode in Bibupload

[Piotr Praczyk]

Sorry for spamming, but I think one thing is not working correctly or I dont 
understand something


Following the instructions in bibupload admin guide, I wanted to remove a 
bibdocfile:


http://www.loc.gov/MARC21/slim";>
1182112

   Data
   .txt
   DELETE-FILE





after running bibupload on this file, (--correct mode) the file is still 
referenced form MARC XML btu when I try to retrieve it, there is 500 error 
produced by the seerver.


I was trying to save the situation by uploading following file (it should 
synchronise MARC XML from the internal data structures) :


http://www.loc.gov/MARC21/slim";>
1182112

FIX-MARC

  



but there was no effect



cheers
Piotr

RE: [INSPIRE-DEV] Why I do not use regression tests as often as I should

[Piotr Praczyk]

Hello

As it seems that everyone who answered agrees with the idea that we need to 
distinguish should-fail tests from those that fail because something is broken, 
may I propose using the 
"expectedFailure" decorator for marking all tests that are not passing at 
current stage of the development ? 

This decorator is available by default in Python higher than 2.7 and I can 
write a small library that will provide this decorator in older installations.

Example:

class ExpectedFailureTestCase(unittest.TestCase):
@unittest.expectedFailure
def test_fail(self):
self.assertEqual(1, 0, "broken")

This should address all the problems that were mentioned in previous e-mails.

More documentation:

http://docs.python.org/library/unittest.html

Piotr  

From: Jan Age Lavik
Sent: 20 July 2012 11:53
To: Alessio Deiana
Cc: Piotr Praczyk; Samuele Kaplun; project-invenio-devel (Invenio developers 
mailing-list); inspire-dev
Subject: Re: [INSPIRE-DEV] Why I do not use regression tests as often as I 
should

Hello guys,

Interesting discussion of a topic that I think deserves one. As all of
you surely are, I am in favor of introducing more testing, unit-tests or
regression, overall in Invenio (and INSPIRE). It makes for better
quality of life :-)

There are many small things that have bitten us because of lack of test
coverage, and as Piotr touches upon, if one wants to change something
that may have minor or major repercussions one feels almost scared of
the consequences in doing it - meaning that new bugs not covered by
tests appears and breaks the application.

Over on the INSPIRE team, we are planning to introduce our own little
ecosystem of code review and testing requirements before shipping to
production (or master codebase) - in addition to the other best
practices of Invenio development, of course.

For example, we are thinking of requiring the implementation of tests
for bugs that appear and gets fixed as well as for new features before
shipping out (to production or Invenio master). This enforcement, and
making sure the tests make sense, will be part of the responsibility for
the code reviewers which we hope to do much more of in the near future.

We see this kind of set-up is more and more common in other software
projects and we hope it is something we can take advantage of. More on
this later.

On the topic of deliberate failing tests in the codebase: I think these
should definitely be made very clear to be intended to fail, and when
running the normal unit-test suite they should not come up as "normal"
errors/failures - that is potentially confusing and time inefficient as
Piotr points out. In addition, if some test case is deliberately failing
because it should be fixed by someone, I think there are better
alternatives to bring awareness to it. For example ticketing systems
such as Trac.

Cheers,
Jan


On 07/20/2012 10:59 AM, Alessio Deiana wrote:
> On Jul 19, 2012, at 4:48 PM, Piotr Praczyk wrote:
>
>> Hello !
>>
>>> From: Samuele Kaplun
>>>
>>> Hi Piotr,
>>>
>>> In data mercoledì, 18 luglio 2012 11.10:23, Piotr Praczyk ha scritto:
>>>> It is less related to the failing tests themselves, but if the mechanisms 
>>>> of
>>>> testing as quality measure of the software does not work, it does not
>>>> motivate to write new regression/unit tests.
>>> On the other hand some people prefer to follow the test-driven-development, 
>>> by
>>> first implementing tests for functionalities that not yet exists so that 
>>> they
>>> will be eventually implemented as originally designed. YMMV.
>> Up to now, my understanding of test-driven development was slightly 
>> different and I did not even think about such approach.
> What I noticed is that we always test our code when coding. The problem is 
> that test is manual. By taking the time to automate
> it, you get your time back very fast since you are basically running tests 
> all day: adding some coding then testing,
> adding some code, testing etc.
>
>> I always though about writing tests for currently implemented feature and 
>> satisfying them before a commit.
>> I think, the weakness of this approach lies in the size of a team that is 
>> collaborating on a project.
>> If You have small number of frequently communicating developers, this can 
>> work.
>> If everyone starts uploading failing tests, all other developers have to 
>> deal with them and see results (which results in issues described in 
>> previous e-mail).
>> It is very difficult to distinguish tests that should fail because things 
>> are not implemented from these that fail because something is broken.
>> Moreover, commiting failing tests carries a risk of commi

RE: [INSPIRE-DEV] Why I do not use regression tests as often as I should

[Piotr Praczyk]

Hello !

>From: Samuele Kaplun
>
> Hi Piotr,
>
>In data mercoledì, 18 luglio 2012 11.10:23, Piotr Praczyk ha scritto:
>> I think we are missing a consistent testing policy.
>>
>> As a developer, I don't feel very motivated to write regression/unit tests
>> if every time I try to run regression tests on the master branch  (which is
>> supposed to be clean), I can see number of errors suggesting that half of >
>> Invenio is broken.
> That's way a strong statement, isn't it? Not taking into account web tests so
> far we have 616 successful unit tests out of 616 (!) and 479 successful
> regression tests out of 490(!), so I don't see Invenio that broken in that
> sense.

Indeed, the number is smaller, but I was  talking about a general impression 
after 
seeing several pages of error messages rather than exact numbers.

>> The most obvious problem with current situation  is that potential errors
>> which were supposed to be tested by a broken test can not be detected.
>> This is particularly annoying when one modifies an important feature and
>> sees errors related to the test output.
>
>Good point.
>
>> There are also some a bit less obvious problems.
>> The difference between having no errors in the output and having one (or
>> more errors) is huge and alarms the developer that her code is broken. If 11
>> tests fail instead of 10, the difference is small and it is much easier to
>> ignore the warning.
>> Another problem is that the productivity of developers decreases if they
>> have to go over and over the same error list (usually being verbose and
>> requiring knowledge about the test source code to understand) searching for
>> differences between previous error messages and new ones.
>
>That is made less problematic by Tibor's development script which will display
>the diff of new kwalitee problems and new regressions.
>
> <https://github.com/tiborsimko/invenio-devscripts>

I did not know about this feature of kwalitee  I was only using it as a 
wrapper around pylint

>> It is less related to the failing tests themselves, but if the mechanisms of
>> testing as quality measure of the software does not work, it does not
>> motivate to write new regression/unit tests.
>
>On the other hand some people prefer to follow the test-driven-development, by
>first implementing tests for functionalities that not yet exists so that they
>will be eventually implemented as originally designed. YMMV.

Up to now, my understanding of test-driven development was slightly different 
and I did not even think about such approach.
I always though about writing tests for currently implemented feature and 
satisfying them before a commit.
I think, the weakness of this approach lies in the size of a team that is 
collaborating on a project.
If You have small number of frequently communicating developers, this can work. 
If everyone starts uploading failing tests, all other developers have to deal 
with them and see results (which results in issues described in previous 
e-mail).
It is very difficult to distinguish tests that should fail because things are 
not implemented from these that fail because something is broken.
Moreover, commiting failing tests carries a risk of commiting tests which will 
never succeed because they are simply not well written (This is the case with 
at least one currently commited test).
This also leads to trouble.
Maybe it is better to use task tracking system for new features or at least 
mark tests as not-satisfied, so that testing framefowk could distinguish them 
automatically ? 
I guess having a separate repository for tests of future features (one branch 
related with one ticket) is a bit of an overkill on the side of infrastructure, 
but we could think about something better than now.

>> I realised I stopped running regression tests at all knowing that every time
>> I install a brand new Invenio, I have a lot of errors.
>
>In general we strive for having unit/regression tests being all successful for
>stable releases. The current situation is that unit test are all working
>nicely, while some regression tests are failing namely in the search syntax
>extension developed within INSPIRE, precisely due to the above TDD. (For
>BibClassify I instead don't know the reason for the failure).

Is't it against the principle of having commits describing complete features ? 
Brokebn tests = broken/missing features.
If I understand correctly, this principle lies in the basis of restrictive 
commit policy and having commit rights by only very limited number of people ?

>Also some time regression tests have been introduced to underline the
>existence of a bug, so that eventually someone would feel responsible for
>addressing 

RE: [inspire-dev] Several questions about search engine of Invenio and possible extensions

[Piotr Praczyk]

Just one more thing 

>>IMHO, there is a philosophical issue here. In Invenio, up to now, the
>>unit of information has always been the record. Everything is built and
>>optimized around it.

>I am completely aware of this, but at the moment, we decided that we do not 
>want to upload figures as records,
>we decided that we have to provide enough infrastructure for non-marc entities.

If we wanted to have all meta-data in MARC records, and records for all 
entities, we would not need any major extensions to the BibDoc infrastructure, 
on which I am working since a while.
We would also not need anything like partial METS support.

cheers
Piotr

RE: partial METS support for storing figures

[Piotr Praczyk]

Hi Sam :)

I do not have any code related to METS support yet.
I am reorganising the branch with BibDocs (as ideas evolved a bit) and soon I 
will push it.

cheers
Piotr

From: Samuele Kaplun
Sent: 29 June 2011 17:52
To: Piotr Praczyk
Cc: project-cdsware-developers (CDS Invenio developers); Salvatore Mele; 
Francisco Javier Nogueras Iso
Subject: RE: partial METS support for storing figures

Il giorno mer, 29/06/2011 alle 15.44 +0200, Piotr Praczyk ha scritto:
> Hello
>
> In this case of no controversies, I am going to implement this schema.
>
> With the plugin infrastructure loading different BibDoc classes
> depending on a type (discussed in different discussion), we can even
> allow different meta-data formats inside METS.
>
> cheers
> Piotr

Hi Piotr,

unfortunately I had not yet found time to analyze and understand your
METS proposal, but anyway, provided you retain feature-backward
compatibility, please go ahead :-) (can you push any early code you have
developed up to now, to a public place so that I can see it?)

Cheers!
Sam

--
Samuele Kaplun
Invenio Developer ** <http://invenio-software.org/>

RE: [inspire-dev] Several questions about search engine of Invenio and possible extensions

[Piotr Praczyk]

Hi


>Il giorno mer, 29/06/2011 alle 12.47 +0000, Piotr Praczyk ha scritto:
>> Some time ago Tibor mentioned the possibility of enabling the BibIndex
>> to take into account some fields from MoreInfo entries in BibDoc.
>> There was also a discussion about possibility of splitting namespace
>> of records into serveral independent chunks (for example having
>> records from 10^8 - 10^12  be assigned to different class of records).
>> These two features would be very useful for the use case of data
>> preservation in Inspire (and also figures infrastructure would benefit
>> from them).
>Yup! Another nice discussion for project-cdsware-developers :-)
So I CC the group


>IMHO, there is a philosophical issue here. In Invenio, up to now, the
>unit of information has always been the record. Everything is built and
>optimized around it. 

I am completely aware of this, but at the moment, we decided that we do not 
want to upload figures as records,
we decided that we have to provide enough infrastructure for non-marc entities.

>BibIndex creates indexes where to
>words/pairs/phrases are pointing to an intbitset containing record ids.
>An intbitset, altough extremely fast, when loaded in memory will fill up
>as many bytes as the biggest ID stored, divided by 8. So if you plan to
>store IDs such as 10^8 this would lead to enormous intbitsets, actually
>using at least 12.5MB each, even in the case there is only 1 ID.

10^8 is completely reasonable number (for example when talking about figures).
Nonetheless, I think the situation is not that bad. If there was splitting of 
namespaces, BibIndex could create intbitsets for separate "namespaces" (ranges 
of ID's).
In this case we would have the biggest intbitset of the size of the biggest 
cluster / 8.

>> We need to be able to store custom objects that are not attached to
>> publications. (Another discussion about technicalities of extending
>> BibDocFile infrastructure). There objects would be prefect for storing
>> data files that should be preserved.
>> The objects should be searchable and have their splash pages (that
>> could be registered in the organisation that manages DOIs)
>If they are searchable doesn't it mean they have metadata? Aren't in
>that case worth having a record representing them?

Indeed they have meta-data, but first, we do not want to show all the metadata 
(could be solved by hiding fields).
Second, the meta-data is not natively MARC so we would have to provide 
transformations both ways between for example DataCite and MARC.
Moreover we would have to standarise MARC. (I am scared of even thinking of 
this as most of datafieds we need are not predicted in the standard)

Third, the MoreInfo is effectlively sotring Meta-Data and we were talking with
Tibor about enabling the possibility of indexing some fields in the future. The 
situation is not much different here.

>> As far as I know, BibIndex creates simple maps word-> set of record
>> numbers and record number-> set of words.
>> Do You think, it would be possible not only to split namespaces but
>> also allow some numbers point to non-record entities ?
>> We could for instance have record numbers 10^12 till 10^123 pointing
>> to custom objects. There would be a separate splash page for these and
>> search engine could be easily integrated.
>> While showing the search results we could simply look at the number
>> and if it belongs to a certain range, link to /object/12345 rather
>> than /record/12345.
>>
>> Another solution (I imagine, much more difficult), would be to change
>> BibIndex to create maps pointing to a bit more structured identifiers
>> -> r1234 is would be record 1234 and o1234 would be object 1234.
>In both cases, the only solution then would be to abandon intbitset
>which would imply a loss of 10 to 100 times in performance.

I think, it resembles the integration of Lucene by Roman ? 
Merging results from intbitsets and external sources sounds familiar.

>> What do you think about this ? This would allow much clearer treatment
>> of custom objects as they would not have to contain two different
>> entities in Invenio - record and BibDocument (or later BibObject).
>>
>> >From the code point of view, we could have a splash page for a
>> BibDocument (BibObject). These splash pages could be aggregated in
>> record display tabs in the case of
>> havni objects attached to regular bibliographical records.
>>
>>
>> The second considered solution involves having MARC records for every
>> BibObject, but this seems to be a bit cumbersome for several reasons
>>  * The number Invenio entities describing the same object would
>>grow

>What exactl

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

> Do you know about the pluginutils class?

Thanks ! I was not aware of this. I will use it.

> Maybe we can provide BibDoc
> classes as plugins that will be automatically instantiated based on the
> new column in bibdoc.
>Maybe we can have a new column in the bibdoc table to say which class
>should be used?

Can not this be simply a document type ? 
Every plugin could provide a boolean function taking type and saying if this 
type is supported.
Besides this, a class or a factory method creating bibdoc could be available.

>> and in addition, some general flags are vert much BibDoc related
>Yes, indeed these flag were introduced to aid certain future workflows
>but I guess they will be easily superseded by the extended
>functionalities of MoreInfo.
Currently they are stored inside MoreInfo.
Here I will try to port existing code rather that rewriting the flags subsystem 
which seems to be independent on the model... just some flags should be 
probably supported by subclasses of BibDoc (BibObject)

>> ... and besides, maybe we should move methods responsable for
>> displaying bibdocfiles to something like template ? Currently there are
>> functions displaying HTML code inside the BibDoc (and similar) class.
>Agree.
done

Besides, this part could be also supported by the same plugins. We might want 
to provide templates based on type.

cheers
Piotr

RE: partial METS support for storing figures

[Piotr Praczyk]

Hello

In this case of no controversies, I am going to implement this schema.

With the plugin infrastructure loading different BibDoc classes depending on a 
type (discussed in different discussion), we can even allow different meta-data 
formats inside METS.

cheers
Piotr

From: Piotr Praczyk
Sent: 24 June 2011 19:51
To: project-cdsware-developers (CDS Invenio developers); Salvatore Mele; 
Francisco Javier Nogueras Iso
Subject: partial METS support for storing figures

Hello

Last week I had some other things so I did not manage to do a lot towards the 
BibDoc generalisation.
Could You please look at this and tell if it makes sense (from 2 persectives:
   1) the consistency of the proposition
   2) The compliance with METS standard )

Implementation of this will make Invenio input METS-like, but it will not allow 
input from other repositories.
It is very very unlikely that data coming from other repositories will be 
formatted according to this schema so maybe it is better not to make it 
METS-like
so that nobody gets confused that gerneral METS is supported ?

Do I understand correctly that every METS document is supposed to describe 
exactly one object ?  This would make the example break conventions of METS.

Here how I imagine mapping of described earleir objects:

- The structMap being a container for objects (every top-level division would 
be interpreted as one of  entities described in one of previous e-mails)

- Uploading/modifying Bibobjects
A division having a type "BibObject" would represent a single BibObject.

The ID field can be used for encoding versions, permanent ids and 
temporary ids (depending on prefixes)

BibObject division should not contains subdivisions, only  links 
to files appearing in BibObject.

example:

   
 


 
   


or new version of an existing record:

  
 

 
   


- relations between versions of BibDocs implemented using structLink element

for example:

Linking using temporary identifiers:
   


Linking already existing entities:
   


Similarly, we could provide relations between particular bibdocfiles.

- storing more-info in Descriptive metadata objects in "binary" format 
(serialised JSON) and assigning using DMDID to entities.
  This would be simple to implement but clearly would make this part of METS 
files non-readable


I think, we should go also for described earleir additional MARC field.


Thanks for comments.

cheers
Piotr

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello


I have also removed some functions from bibdocfile (they were never used)
Just in case anyone needs them, I provide a list:

toplevel functions:

generic_path2bidocfile
normalize_version
compose_file
bibdocfile_url_to_fullpath
get_docid_from_bibdocfile_fullpath
decompose_bibdocfile_old_url
get_format_from_url
readfile


BibRecDocs class:

get_xml_8564 (used only in regression tests)


cheers
Piotr


From: Piotr Praczyk
Sent: 28 June 2011 00:26
To: Samuele Kaplun
Cc: Tibor Simko; project-cdsware-developers (CDS Invenio developers); Salvatore 
Mele
Subject: RE: [inspire-dev] Limitations with having standalone BibDocs

Hello

Just a small part missing from previous discussion


>> - BibDoc Python class in fact reflects needs of FullTexts. It should
>> probably be stripped from functions that are typical to FT treatment.
>> They should be moved to a subclass. (ie functions extracting fulltext)

>Well it represent a document. Could you be more specific about which
>functions are only related to fulltext?

get_text
get_text_path
extract_text
pdf_a_p
has_text


and in addition, some general flags are vert much BibDoc related


... and besides, maybe we should move methods responsable for displaying 
bibdocfiles to something like template ?
Currently there are functions displaying HTML code inside the BibDoc (and 
similar) class.


cheers
Piotr

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello

Just a small part missing from previous discussion


>> - BibDoc Python class in fact reflects needs of FullTexts. It should
>> probably be stripped from functions that are typical to FT treatment.
>> They should be moved to a subclass. (ie functions extracting fulltext)

>Well it represent a document. Could you be more specific about which
>functions are only related to fulltext?

get_text
get_text_path
extract_text
pdf_a_p
has_text


and in addition, some general flags are vert much BibDoc related


... and besides, maybe we should move methods responsable for displaying 
bibdocfiles to something like template ?
Currently there are functions displaying HTML code inside the BibDoc (and 
similar) class.


cheers
Piotr

partial METS support for storing figures

[Piotr Praczyk]

Hello

Last week I had some other things so I did not manage to do a lot towards the 
BibDoc generalisation.
Could You please look at this and tell if it makes sense (from 2 persectives:
   1) the consistency of the proposition
   2) The compliance with METS standard )

Implementation of this will make Invenio input METS-like, but it will not allow 
input from other repositories.
It is very very unlikely that data coming from other repositories will be 
formatted according to this schema so maybe it is better not to make it 
METS-like
so that nobody gets confused that gerneral METS is supported ?

Do I understand correctly that every METS document is supposed to describe 
exactly one object ?  This would make the example break conventions of METS.

Here how I imagine mapping of described earleir objects:

- The structMap being a container for objects (every top-level division would 
be interpreted as one of  entities described in one of previous e-mails)

- Uploading/modifying Bibobjects
A division having a type "BibObject" would represent a single BibObject.

The ID field can be used for encoding versions, permanent ids and 
temporary ids (depending on prefixes)

BibObject division should not contains subdivisions, only  links 
to files appearing in BibObject.

example:

   
 


 
   


or new version of an existing record:

  
 

 
   


- relations between versions of BibDocs implemented using structLink element

for example:

Linking using temporary identifiers:
   


Linking already existing entities:
   


Similarly, we could provide relations between particular bibdocfiles.

- storing more-info in Descriptive metadata objects in "binary" format 
(serialised JSON) and assigning using DMDID to entities.
  This would be simple to implement but clearly would make this part of METS 
files non-readable


I think, we should go also for described earleir additional MARC field.


Thanks for comments.

cheers
Piotr

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

>>I thought, we do not want to have a meta-data about figures in MARC ... 
>>rather standalone BibDoc remembering everything, but if we wanted to create 
>>one record per figure, indeed this would be solved.
>>(Again brings the question if we want to multiply our record space by 3)

>...that's MARC multilevel description! (in MARC21 dialect: 
>http://www.loc.gov/marc/bibliographic/bd76x78x.html)

If You mean referencing MARC records of figures from the main article document, 
I think there were some more pragmatic issues with this

- the Invenio ability of dealing with ~1000 records (The number of records 
would be around this)
- separation of records describing publications and other objects (In Inspire 
we currently have records only for publications)
- What would  purely figure-MARC store ? All the important information would be 
stored in tie BibObject (earlier BibDoc).

the advantage of having records for figures would be ease of reutilising 
current search metaphores ... though only for textual data.

I will try to answer the longer e-mail soon. (After understanding ... I am not 
a librarian and it is difficult to understand all related standards and aspects 
of them at once)
I would just like to note that in what I understand, a descriptive meta-data of 
figures, which would change just as often as MARC records, would be encoded in 
METS.

I will have a look at eamples you mentioned.


I think, the biggest question at the moment is: Do we need to change the 
proposed data-model (Bibobject, BibFiles etc...) to allow importation of METS 
descriptions.
I can see several issues with this, for example we do not allow arbitrarily 
nested divisions. just document->files (with the dimension of versions).
How should we deal with custom meta-data ? you can enclose arbitrary RDF inside 
METS, while we were so far talking about a dictionary of depth 2 structure.


Piotr

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Just a small reflection on METS

If we had a support for it (in particular the structural meta-data, we could 
for example synchronise transcript of the fulltext document with the internal 
structure of a printer-ready version (by remembering division into pages / 
chapters etc... with different types of pointers)
This could be very useful for many purposes (including treatment of figures)

cheers
Piotr 

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello


I have noticed, that I have repeated some parts of your answer in my previous 
e-mail.


From: Samuele Kaplun
Sent: 13 June 2011 21:43
To: Piotr Praczyk
Cc: Tibor Simko; project-cdsware-developers (CDS Invenio developers); Salvatore 
Mele
Subject: RE: [inspire-dev] Limitations with having standalone BibDocs


>> not necessarily. the use cases I know (probably there are more about which 
>> Salvatore and Suenje have knowledge) are :
>> 1) (the most obvious for me) - the case of standalone plots showing an 
>> important phenomenon. The access to them shall be provided by the figures 
>> search.
>Isn't this addressed by associating a record to the standalone plot?
>(especially to search for it, I guess you will do it through its
>metadata).

I thought, we do not want to have a meta-data about figures in MARC ... rather 
standalone BibDoc remembering everything, but if we wanted to create one record 
per figure, indeed this would be solved.
(Again brings the question if we want to multiply our record space by 3)


Piotr
--
Samuele Kaplun
Invenio Developer ** <http://invenio-software.org/>

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello

Thanks for Your answer.
As I mentioned before, my knowledge of METS is very very limited (to one look 
into the tutorial), so your comments are very precious..
My main concerns with implementing the standard is its compatibility with our 
data model (which is changing anyway, so if we are to support METS, it is 
better to rethink most of things now and possibly support part of the standard).
The  second concern was the amount of effort needed to implement [but this is 
irrelevant for the discussion].
Indeed it is obvious, how beneficial it would be to have a full support for the 
standard.

>> 1) (the most obvious for me) - the case of standalone plots showing an 
>> important phenomenon. The access to them shall be provided by the figures 
>> search.
>I propose the benchmark of document digitization use-case: let's think about
>- one document with its metadata (say MARC or EAD record)
>- with many image files (or video/audio files...) for the different pages (or 
>streaming parts...) with many different file-versions per page (like: the 
>master uncompressed version for archive; the high-level compressed version for 
>local access; the low-level compressed version for online access; the 
>thumbnail; etc etc).

This is not the use case of figures from scientific publications (If I 
understand correctly, looks rather like digitalisation of entire documents), 
though seems to be relevant for Invenio/Inspire in general. Looks like a nice 
benchmark of the underlying data-structures.

>>>  FFT should be left for fulltext upload where it serves the purpose
>>>  perfectly and should be understood as syntactic sugar providing 
>>> abbreviated
>>> form of a more general upload.
>>It also serves well the case of many documents in many formats attached
>>to the same records. I hope all these use cases will still be supported
>>through FFTs
>The idea was to provide new mechanism not modifying the existing capabilities 
>of Bibupload... just stopping to use FFT for objects as Figures.

Exactly. The FFT syntax should be an abbreviated form of attaching documents, 
equivalent to some (METS?) input. 


>>  * provide a web handler to access bibdocfiles regardless of them
>>being owned by a record (as the
>>current /record/123/files/foo.pdf will no longer work for non
>>fulltext) (BTW what about restriction/authorization? What if
>>bibdoc is referenced both by a public and a restricted record?
>>Should we go for the strongest restriction mode?)
>It canstill work, but in a slightly more distant future we might want to 
>provide /object/123 along with /record/12

>Here (with this ad-hoc selection of quotes) I'd like to forward a big 
>preoccupation of the system analysts I worked with for digital libraries: the 
>complete autonomy of the storage file-system from metadata file-pointers.
>Digital-metadata like METS are often used as a layer (additional to the 
>descriptive-metadata like MARC) to store file-pointers managed with some 
>resolver (based on algorithms like yours). So that when we are dealing with 
>many many thousands of files per many many TeraBytes, EVERY >new 
>accommodation/substitution/refresh of the storage-system is possible … without 
>worrying about the logical (even permanent) pointers to the files. (From METS 
>tutorial: The LOCTYPE attribute specifies the type of locator contained in 
>body of the element; valid values for LOCTYPE include >'URN,' 'URL,' 'PURL,' 
>'HANDLE,' 'DOI,' and 'OTHER.')

>Speaking in concrete words: in my experience quite every time I saw,
>- descriptive-metadata (like MARC) managed on one side with specific 
>(multiple) identifiers (..also modifiable identifiers, in the collaborative 
>systems..)
>- digital-repositories, on the other side, with specific (stable!!) 
>identifiers for digital-objects and their component files,
>- and, in the middle, digital-metadata (like METS) which guarantee the 
>connections (regardless the physical file storage).


I think, I did not understand this part.
What are the cases of modifiable identifiers inside MARC ? Titles of documents 
+ authors ? Exact file paths in the file system (as we happen to still have in 
some places in Inspire ?) 
By link between two do you mean a document identifying the same document with 
both at the same time ? 
What is physical storage for You ? From the physical storage I wanted to 
abstract exactly by providing such links /object/DOCID


>>BDR is supposed to provide link between records and objects(document).
>>In METS (if I understand correctly), they are used only to describe the 
>>internal structure of objects.
>I'm not sure if I'm correctly following your arguments, but here I have to 
>suggest just the contrary: METS does provide link between records and 
>objects(document). And internal structure of objects. And technical 
>description of the digital-object...
>METS is a container which can include or point-to a lot of different layers 
>with 

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello !

I have never heard about METS before so today I invested some time to learn 
about it (Which means, I have some but rather limited knowledge).
>From what I have read, I have an impression that the model described by METS 
>is in some areas much more detailed (For  that the one we are using (with 
>extensions)
In some other areas, it is missing concepts we need.

What level of support for METS did you have in mind ? Exporting data. importing 
in full format, importing in some subset of the format ? 
Do You think, we could easily implement something that would deal with concepts 
present there ? 
   - metadata of objects encoded in different formats (it looks like a more 
general concept of MoreInfo)
   - relations between different parts of different files (in the examples, 
they showed how to divide monolith files into chapters and link them between 
iach other (audio format with transcript format))
Implementing all this sounds like a rather big project for me ... especially 
that we would have to extend the standard (which would make us again not 
completely compatible).
What do you think about this ? Maybe METS should not be native to Invenio but 
we should start with supporting the possibility to export data in this format ? 



> What in the end is the use case of standalone documents? How
> can you later search for them? I guess they make sense only if they are
> at the same time referenced by at least another document or by a record,
>isnt't it?

not necessarily. the use cases I know (probably there are more about which 
Salvatore and Suenje have knowledge) are :
1) (the most obvious for me) - the case of standalone plots showing an 
important phenomenon. The access to them shall be provided by the figures 
search.
2) The case of data preservation towards which Inspire is turning - useful 
files of experimental data (usually they will be attached to papers but not 
necessarily)

>>  Of course, it is possible to encode anything in MARC, but it will
>>  quickly become unreadable and the code implementing encoding/decoding
>> of the data will be more error-prone.
>>  FFT should be left for fulltext upload where it serves the purpose
>>  perfectly and should be understood as syntactic sugar providing 
>> abbreviated
>> form of a more general upload.

>It also serves well the case of many documents in many formats attached
>to the same records. I hope all these use cases will still be supported
>through FFTs

The idea was to provide new mechanism not modifying the existing capabilities 
of Bibupload... just stopping to use FFT for objects as Figures.




>> + Uploading of the documents
>>
>> Current mechanism for uploading documents to Invenio is very much
>> oriented towards managing fulltexts that can belong to only one record.
>> It is difficult to extend BibUpload to allow attachments of the same
>> BibDoc to many records or to create BibDocs not related to any record
>> using the FFT syntax.

>This make me think we should not probably use BibUpload to manage
>BibDocs (beside keep on with the FFT thing). What we have done
>up-to-now, for very complex manipulation of BibDocs (e.g. in case of
>WebSubmit), was to do everything with the API, and then send an FFT with
>a FIX-MARC to synchronize the 8564_ fields so that they reflect the last
>state of the documents.

>So you would use it only for documents that are attached to at least one
>record, isn't it? Otherwise you should really think of implementing a
>tool separated from BibUpload that can act irrespectively of records.

Indeed, I was thinking about having one tool only because of the temporary 
identifeirs described later.
Having one tool makes their semantic easier. We do not have to implement 
complicated scenarios of detecting if a given temporary id is still necessary.
Do you see any elegant solution to this problem with separate tasks ? 


>In particular its "File Section"
> would
>match the current FFT
>and the "Structural Map"
>
>and the "Structural Links"
>
>do really sounds to me as your BDR proposal.

BDR is supposed to provide link between records and objects(document).
In METS (if I understand correctly), they are used only to describe the 
internal structure of objects.


>> In addition to extending the syntax of BibUpload, the significance of
>> internal BibDoc identifiers should be increased. It should be assured
>> that the same identifier can not be reused after deletion of a BibDoc.
>Are you talking about docnames? 
I was rather thinking about BibDocId (represented in the database and used 
internally by some parts of the API)


>> +++ 
>>
>> This markup element enables uploading links between BibDocs being uploaded 
>> to Invenio or already existing
>>
>>
>> Example:
>>
>> > version2="2" type="extracted

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello. 

This is a follow-up to the previous E-mail about BibUpload.
Here, I present I think the simplest to implement method of extending BibUpload 
to fit new requirements (despite long description it should be short to 
implement because most of work is already done or can be redirected to external 
libraries).
Because proposed extensions touch a very core functionality of Invenio, I would 
be very happy having green light to do this.
This part of Invenio is also crucial for my project which can not progress much 
before having these parts integrated in Invenio codebase.

As I mentioned, I have a branch containing implementation of some new features 
of BibDoc - mostly extension of the MoreInfo notion and introduction of 
relations between records.
The main point of this e-mail is ingestion of new documents to Invenio. Main 
ideas inspiring the work are STANDALONE BIBDOCS, RELATIONS BETWEEN ONE DOCUMENT 
AND MANY RECORDS and UPLOAD OF CUSTOM DATA INTO MOREINFO STRUCTURES.
I believe the proposed schema is flexible, but I am not able to predict all 
possible scenarios, so critical comments (scenarios in which something could 
break), especially before the phase of implementation would be very much 
appreciated.

Please excuse the length of the e-mail. I was trying to keep everything clear 
and as short as possible, yet providing examples which might be useful in 
understanding the idea.
In the ideal situation, I would like to have an implemented prototype of 
similar (modified after comments) infrastructure during the Inspire week which 
will take place in the beginning of July.


+ Reasons, why current upload system is not sufficient and we should rather 
provide a new mechanism rather than extending the existing one
  - Using BibUpload "MARC" is not a very clean solution for uploading
non-record data such as standalone documents.
In order to be compliant with the current BibUpload format, We would
have to include a FFT tag inside a record tag.
This would have to be interpretet by BibUpload as NOT modifying or
uploading any record.
 
   - Packing data of a more complicated structure into "MARC" is non-intuitive.
 Of course, it is possible to encode anything in MARC, but it will
 quickly become unreadable and the code implementing encoding/decoding
 of the data will be more error-prone.
 FFT should be left for fulltext upload where it serves the purpose
 perfectly and should be understood as syntactic sugar providing 
abbreviated 
 form of a more general upload.

   - FFT stands for Fulltext File Transfer. Using it for non-fulltext
 docuemtns leads to confusion. 

   - The BibUpload "MARC" obfuscates the way of thinking about documents.
 The internal structure of documents and relations among them (and relations
 between documents and records) is not reflected in the structure of the 
FFT field.

 Portions of information from different subfields land in completely
 different database entities.
 Example: Type of document. Even the BibUpload documentation suggests
   that is a tyope of document while in fact, it is a type of link
   between document and a bibliographical record. Currently it is
   equivalent, but it will not be after we allow multiple record
   attachemnts.


+ Uploading of the documents

Current mechanism for uploading documents to Invenio is very much oriented 
towards managing fulltexts that can belong to only one record.
It is difficult to extend BibUpload to allow attachments of the same BibDoc to 
many records or to create BibDocs not related to any record using the FFT 
syntax.
It is also difficult to provide uploading of relations between existing 
BibDocs. This is because MARC provides method of encoding tree-structured data 
with the  maximum nesting depth of 2. (1 in the case of special fields). Data 
structures that need to be uploaded to Invenio are graphs.
BibUpload is the only gateway for uploading data to Invenio. For the sake of 
uniformity, it should remain this way.

I believe, the easiest and the most efficient way of adapting BibUpload to 
extended usage of BibDocs and file attachments is prviding a non-MARC XML based 
additional input that could be processed by BibUpload.
The existing FFT (Fulltext File Transfer) field should be preserved and 
utilised ONLY when uploading fulltext documents.
FFT should be understood as a convenient abbreviated syntax allowing a limited 
functionality of new XML syntax.
In addition to FFT, a new "MARC" field could be introduced - BDR (BibDoc 
Reference). 
The purpose of this field would be to introduce a link between uploaded (or 
modified) record and an existing BibDoc.
This would enable multiple relationship between records and documents which is 
currently explicitly blocked in the code of BibDoc class but theoretically 
allowed in the database structure.

In addition to extending the syntax of BibUpload, the significance of internal 
BibDoc identifie

RE: [inspire-dev] Limitations with having standalone BibDocs

[Piotr Praczyk]

Hello

I have a branch, but it is still in my private repository. I will push it to 
public hopefully today.
(I am struggling with weird problems with regression tests... was 
BibRecDocsTest.test_BibRecDocs ever passing? For my taste the test requests 
incorrect file sizes ... and indeed it fails on my machine)

Indeed a type might be important for the connection between record and 
document. In Your example though, the object is plot in both cases. 
In one, also has the function of being part of a bigger record and in the 
second case, the record describes a plot itself, so there is a sense in having 
type stored in two places  - in BibDoc and in the relation to a record as it 
means something slightly different.

Still, there is a problem, how BibUpload should deal with non-record data and 
with relations between documents. 
The document identifier is internal to Invenio and should not be passed inside 
"MARC" FFT (or some similar field). Addressing by recordId/Name seems not to be 
universal enough if we generalise the BibDoc file.


Piotr


From: Samuele Kaplun
Sent: 06 June 2011 14:18
To: Piotr Praczyk
Cc: inspire-dev
Subject: Re: [inspire-dev] Limitations with having standalone BibDocs

Hi Piotr,

Il giorno lun, 06/06/2011 alle 11.57 +0000, Piotr Praczyk ha scritto:
> Some time ago I was talking with Tibor about a possibility of having
> standalone BibDocs.

I think it might be better to move this discussion on
 mailing list as your issue are really
touching one of the core parts of Invenio.

> Also Salvatore said that they could be very useful in the near future
> for storing different pieces of data.

Sure they are.

> While going through the source code of bibdocfile.py, I discovered two
> things that are not exactly as they were described to be.
> In particular, the possibility of having a BibDoc not attached to any
> bibrecord is explicitly blocked in the source code of BibDoc class.
>
> (The constructor tries to retrieve the identifier of the record to
> which the document is related and the case of failure, throws one of
> exceptions:
>raise InvenioWebSubmitFileError, "The docid %s associated with
> docid %s is not associated with any record" % (main_docid, docid)
>raise InvenioWebSubmitFileError, "The docid %s is not associated to
> any recid or other docid" % docid
> )

Indeed. On the other hand, since the very link is stored in one simple
table this limitation should be easy to remove. However there are
several assumptions made in several part of Invenio about this link,
namely in the bibdocfile CLI, and as you mention below in BibUpload too.

Another enhancement I was always thinking to add, is the possibilities
to have one bibdoc attached to several records (in the end the
bibrec_bibdoc table allows for a many-to-many connection).

> Moreover, BibDoc does not really hold a type.
> The type of document is a property of the link between record and the
> document.
> Should we modify this ?  This would have some deeper implications for
> BibUpload as we would have to have a possibility of uploading data not
> being associated with a record.

This is also why I would suggest to talk about this in the wider mailing
list. Today the doctype is not such a used property, although it is used
more and more in INSPIRE. On the other hand moving it from being a
property of the connection between a bibrec and a bibdoc, to be a
property of a single bibdoc, would imply that a bibdoc is intrinsically
of a given type, regardless of its context. That means that it would no
longer be possible, in case we allow to have a bibdoc to be pointed by
many records, to sports different types.

A "Plot" within record A can in principle be thought as a "Main" in
record B.

This open a big discussion about support for compound digital objects,
that, since you propose to extend bibdocfile, it would be great to fully
address in the best manner (e.g. taking in consideration METS, and
OAI-ORE use cases).

Shall we build a task force on the topic?

Cheers!
Sam

P.s. on an other subject, we are accumulating use-cases for your
extended MoreInfoBibDocFile, to be able to attach properties either at
the level of BibDocFile or at the level of the BibDoc. Have by chance
already started working on it and do you have a branch with some draft
to play with?


--
Samuele Kaplun
Invenio Developer ** <http://invenio-software.org/>

Re: Invenio Developers Forum - today 11:00 at CERN Library(?)

[Piotr Praczyk]

Hi

I booked the room A next to the main auditorium 61-1-017


cheers
Piotr

2010/5/10 Tibor Simko 

> Hello:
>
> The next Invenio Developers Forum will take place today starting at
> 11:00 at a location close to CERN Library(?).  (GS guys, please let us
> know where, otherwise we can take again 31-S-023 over here.)
>
> There is nothing particular on the agenda today.  We can chat in smaller
> groups about the things we are working on, the branches to get
> integrated, devote time to make a common collaborative sprint to
> complete the documentation, to improve the code kwalitee, to crack some
> minor v1.0 tasks, etc.
>
> Best regards
> --
> Tibor Simko
>

Re: Error 404 for jqueryui

[Piotr Praczyk]

Hi Benoit

I am using JQuery 1.7.3 for BibEdit and everything seems to work fine.
( I do not know, what about other modules).
In my current version I fixed the installation to 1.7.3, but this is
not a long-term solution. ( I will address this issue next)

cheers
Piotr

2010/5/7 Benoit Thiell :
> Hi,
>
> JQuery UI has been upgraded to version 1.7.3. This breaks the "make
> install-jquery-plugins". Can we change this to the new version without
> breaking anything?
>
> Current: http://jqueryui.com/download/jquery-ui-1.7.2.custom.zip
> New:     http://jqueryui.com/download/jquery-ui-1.7.3.custom.zip
>
> Benoit.
>

Re: The Multivio project

[Piotr Praczyk]

Hi !

Looks very interesting :) Thanks for the information about this.
 just as a curiosity... It does not support polish fiscal
declarations ;) (Very few PDF readers do ;) )

http://demo.multivio.org/client/#get&url=http://e-deklaracje.mf.gov.pl/files/pdf/PIT-36%2814%29_v2-0.pdf


cheers
Piotr

2010/5/3 Miguel Moreira :
> i...@multivio.org

Re: JQuery version in Invenio

[Piotr Praczyk]

Hi Sam !

I haven't tried using new JQuery with BibEdit yet, but using the newest
versionis usually a good idea.
What about switching directly after Invenio 1.0 is released ?

Concerning downloading the scrips from a single location. In general I am
for it, but I think it should be configurable.
I can imagine Invenio installations running in some separated network. In
such a case scripts should be hosted too.

cheers
Piotr



2010/4/8 Samuele Kaplun 

> used

[task #14749] Bibmerge crashes on older versions of a record

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #14749, project CDS Invenio

==
 LATEST MODIFICATIONS of task #14749:
==

Update of task #14749 (project cdsware):

 Open/Closed:Open => Closed 


==
 OVERVIEW of task #14749:
==

URL:
  

 Summary: Bibmerge crashes on older versions of a record
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2010-03-29 10:59
 Should Start On: 2010-03-29 00:00
   Should be Finished on: 2010-03-29 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Closed
 Discussion Lock: Any
  Effort: 0.00

___


Viewing older version of a record causes problems with bibmerge.

The first time a revision is opened, everything works fine.
Further accesses cause errors. (This seems to be somehow related to cache
files)



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #14848] BibEdit: private record templates

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #14848, project CDS Invenio

==
 LATEST MODIFICATIONS of task #14848:
==

Update of task #14848 (project cdsware):

 Assigned to:None => piotr  


==
 OVERVIEW of task #14848:
==

URL:
  

 Summary: BibEdit: private record templates
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2010-04-06 13:21
 Should Start On: 2010-04-06 00:00
   Should be Finished on: 2010-04-06 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


There should be a possibility of having private record templates.
Catalogers should be able to save currently edited record as sch a template
and manage existing.



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #14848] BibEdit: private record templates

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #14848, project CDS Invenio

==
 OVERVIEW of task #14848:
==

URL:
  

 Summary: BibEdit: private record templates
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2010-04-06 13:21
 Should Start On: 2010-04-06 00:00
   Should be Finished on: 2010-04-06 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


There should be a possibility of having private record templates.
Catalogers should be able to save currently edited record as sch a template
and manage existing.



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #14749] Bibmerge crashes on older versions of a record

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #14749, project CDS Invenio

==
 LATEST MODIFICATIONS of task #14749:
==

Update of task #14749 (project cdsware):

 Assigned to:None => piotr  


==
 OVERVIEW of task #14749:
==

URL:
  

 Summary: Bibmerge crashes on older versions of a record
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2010-03-29 10:59
 Should Start On: 2010-03-29 00:00
   Should be Finished on: 2010-03-29 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


Viewing older version of a record causes problems with bibmerge.

The first time a revision is opened, everything works fine.
Further accesses cause errors. (This seems to be somehow related to cache
files)



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #14749] Bibmerge crashes on older versions of a record

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #14749, project CDS Invenio

==
 OVERVIEW of task #14749:
==

URL:
  

 Summary: Bibmerge crashes on older versions of a record
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2010-03-29 10:59
 Should Start On: 2010-03-29 00:00
   Should be Finished on: 2010-03-29 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


Viewing older version of a record causes problems with bibmerge.

The first time a revision is opened, everything works fine.
Further accesses cause errors. (This seems to be somehow related to cache
files)



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

Re: Dev forum?

[Piotr Praczyk]

I just tried to book the room C, but it is already taken. I don't know
about any other room that we could use ( besides our office which is
quite big). I will try to ask guys when they return from the coffee.


2010/3/15 Tibor Simko :
> On Mon, 15 Mar 2010, Brooks, Travis C. wrote:
>> Of course I am the worst possible person to do this, but I have
>> possibly secured:
>>
>> The conference room 14-4-030
>>
>> For 11-12:30 today.  If there were/are better options, let me know and
>> I will rescind the booking
>
> Today the forum is happening near the library, so it is usually the GS
> folks who secure a room there.  So let's also check with Piotr et al
> what the plans were...
>
> Best regards
> --
> Tibor Simko ** CERN Document Server ** 
>

Re: Invenio Developers Forum - today 11:00 at CERN 31-S-023

[Piotr Praczyk]

For me both dates are good. Maybe Tuesday 9:00 is a little better ( it
is a little less disorganising to have a meeting in the beginning of a
day rather than in the middle, but both options are appropriate and we
should go for an option convenient for the majority)

>From my pont of view, it is always better to have meeting near my
office ;) ... I suppose some rooms could be already taken.

Do you know the shedule of the Inspire week ? I do not know how both
proposed dates fit with this meeting.

I created a doodle pool, we can vote for the time here :

http://doodle.com/w5nr3x6n3avhz4qs

cheers
Piotr

2010/3/12 Tibor Simko :
> around

Re: Invenio Developers Forum - today 11:00 at CERN 31-S-023

[Piotr Praczyk]

at 11:00 we have a meeting of our section, but maybe after lunch ?

2010/3/9 Ludmila Marian :
> Hi all,
>
> Is there a specific reason for having the meeting on Tuesday? Why not
> changing the day to Friday, and keep the 11:00 ?
>
> Ludmila
>
> Piotr Praczyk wrote:
>>
>> What about even a little earlier ? (9:30 ? )
>>
>> Piotr
>>
>> 2010/3/9 Tibor Simko :
>>
>>>
>>> good for Jerome.  Maybe we can advance to Tuesday 10:00?
>>>
>>>
>

Re: Invenio Developers Forum - today 11:00 at CERN 31-S-023

[Piotr Praczyk]

What about even a little earlier ? (9:30 ? )

Piotr

2010/3/9 Tibor Simko :
> good for Jerome.  Maybe we can advance to Tuesday 10:00?
>

Re: Invenio Developers Forum - today 11:00 at CERN 31-S-023

[Piotr Praczyk]

In the beginning I thought, the change to 11h is temporary. In general
I have  the lessons at this hour regularily, but they end with the end
of march.

Piotr

2010/3/9 Tibor Simko :
> On Tue, 09 Mar 2010, Piotr Praczyk wrote:
>> Sorry, I did not read this e-mail until now ( I was having a French
>> lesson when it was sent). Can we do this for example next week during
>> the meeting ?
>
> Will you be having French lessons regularly on Tuesdays 11:00?  If yes,
> then we may want to shift the meeting time again.
>
> Best regards
> --
> Tibor Simko ** CERN Document Server ** <http://cds.cern.ch/>
>

Re: Invenio Developers Forum - today 11:00 at CERN 31-S-023

[Piotr Praczyk]

Hello

> We try to alternate the location, one week in GS, one week in IT.  Today
> is the turn of the IT.  Moreover, we may be more comfortable in 31-S-023
> in case Piotr projects his web tests on the big screen.

Sorry, I did not read this e-mail until now ( I was having a French
lesson when it was sent). Can we do this for example next week during
the meeting ?

Piotr

Re: What does 'holding pen' mean?

[Piotr Praczyk]

Hi

The concept is about a place that stores content not accepted for the
production database yet and awaiting a manual action (
approval/partial approval or rejection). At the moment it is used
during the OAI harvesting process to put records that have not been
accepted automatically nor rejected. I do not know what word is the
most appropriate in French.

cheers
Piotr

2010/3/2 Ferran Jorba :
> Hello,
>
> sorry to bother, but I don't know what 'holding pen' means, and how to
> translate it into Catalan and Spanish.  Does anybody have a tentative
> translation into similar languages like French, Italian or Portuguese?
>
> Thanks,
>
> Ferran
>

[task #13565] Problem when reloding page under Google Chrome

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #13565, project CDS Invenio

==
 OVERVIEW of task #13565:
==

URL:
  

 Summary: Problem when reloding page under Google Chrome
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2010-01-26 14:23
 Should Start On: 2010-01-26 00:00
   Should be Finished on: 2010-01-26 00:00
Category: BibEdit
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


When reloading the BibEdit page under Google Chorme browser, the search type
gets empty causeing erros in further searches. 

Might be an error in BibEdir or beta version of Chrome ... to investigate 



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11216] BibEdit: field templates

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11216, project CDS Invenio

==
 LATEST MODIFICATIONS of task #11216:
==

Follow-up Comment #1, task #11216 (project cdsware):

published in my public repository ... I hope this will be final version of
this feature

==
 OVERVIEW of task #11216:
==

URL:
  <http://savannah.cern.ch/task/?11216>

 Summary: BibEdit: field templates
 Project: CDS Invenio
Submitted by: simko
Submitted on: 2009-09-14 13:30
 Should Start On: 2009-09-14 00:00
   Should be Finished on: 2009-09-14 00:00
Category: BibEdit
Priority: 7 - High
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


When editing an existing record, if the cataloger wants to add a new
field, s/he can currently add a new empty MARC field only.  For
efficiency purposes, it would be useful to offer the cataloger a
drop-down menu from which s/he could choose the field s/he wants to
add, like:

   Add new field: [empty MARC field]
  [first author]
  [additional author]
  [title]
  [journal publication information]
  [conference information]
  [keyword]

The predefined field templates would be snippets of MARCXML that would
look like:

   
   
   
 
   VOLATILE: Familyname, Firstname
   VOLATILE: Affiliation
 
   

just as the record templates we already provide (see also task
#10260).

The field templates would live in
/opt/cds-invenio/etc/bibedit/field_templates/, a parallel to
/opt/cds-invenio/etc/bibedit/record_templates/.

The existing BibEdit-Template-* headers in XML comments should be
renamed to BibEdit-Record-Template-* to have a nice `record template'
and `field template' naming parallel.

* * *

Feature extension: it would be nice to have a notion of multiplicity
in the field snippet, in order to insert 20 additional authors in one
go, for example to have:

   Add new field: [additional author]
  [5 additional authors]
  [10 additional authors]

where the three field templates would contain different number of
additional authors inside.  In other words, a template snippet could
contain more than one field instance.

Alternatively, one could specify a number of how many times a certain
field template should be applied, like:

   Add new fields: [1] [additional author(s)]
   [2]
   [3]


___

Follow-up Comments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-10-16 16:25      By: Piotr Praczyk 
published in my public repository ... I hope this will be final version of
this feature





___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -COM-
1576| -SUB-




==

This item URL is:
  <http://savannah.cern.ch/task/?11216>

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11314] BibEdit: volatile fields should disappear if not filled in

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11314, project CDS Invenio

==
 LATEST MODIFICATIONS of task #11314:
==

Update of task #11314 (project cdsware):

  Status:None => Done   
Percent Complete:  0% => 100%   
 Open/Closed:Open => Closed 


==
 OVERVIEW of task #11314:
==

URL:
  <http://savannah.cern.ch/task/?11314>

 Summary: BibEdit: volatile fields should disappear if not
filled in
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-24 09:23
 Should Start On: 2009-09-24 00:00
   Should be Finished on: 2009-09-24 00:00
Category: BibEdit
Priority: 5 - Normal
  Status: Done
 Privacy: Public
Percent Complete: 100%
 Assigned to: piotr
 Open/Closed: Closed
 Discussion Lock: Any
  Effort: 0.00

___


When a record is created fgrom the template and volatile fields are not
filled in, their values should disappear rather than be uploaded as
"VOLATILE: field description"

___

Follow-up Comments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-10-16 09:38  By: Piotr Praczyk 
Solved by the commit: c22035dd9e5314d64593d6360bc9f5dd1ba6ca97

The behaviour of volatile fields has changed a little:

   - Every field whose value starts with VOLATILE: will be 
 displayed as such

   - When displaying a volatile field, the VOLATILE: prefix is
 omitted

   - When editing a volatile field, all the content is 
 automatically selected so that the description appearing by 
 default can be replaced easily
  
   - When editing a normal field, the changes are appended at
 the end rather than replacing (the content is not selected)

   - writing VOLATILE: at the beginning of value changes the 
 field to volatile ( with all the consequences such as not 
 displaying the VOLATILE: prefix and changing the css class

   - all the subfields starting with VOLATILE: are omitted when 
 submitting a change


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-24 09:26  By: Tibor Simko 
See also task #10258.  Kyriakos fixed the removing of empty fields, but
VOLATILE fields were introduced later, so the old removing fix is no longer
fully valid.





___

Carbon-Copy List:

CC Address  | Comment
+-
1576| -COM-
5686| -SUB-




==

This item URL is:
  <http://savannah.cern.ch/task/?11314>

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11314] BibEdit: volatile fields should disappear if not filled in

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11314, project CDS Invenio

==
 LATEST MODIFICATIONS of task #11314:
==

Follow-up Comment #2, task #11314 (project cdsware):

Solved by the commit: c22035dd9e5314d64593d6360bc9f5dd1ba6ca97

The behaviour of volatile fields has changed a little:

   - Every field whose value starts with VOLATILE: will be 
 displayed as such

   - When displaying a volatile field, the VOLATILE: prefix is
 omitted

   - When editing a volatile field, all the content is 
 automatically selected so that the description appearing by 
 default can be replaced easily
  
   - When editing a normal field, the changes are appended at
 the end rather than replacing (the content is not selected)

   - writing VOLATILE: at the beginning of value changes the 
 field to volatile ( with all the consequences such as not 
 displaying the VOLATILE: prefix and changing the css class

   - all the subfields starting with VOLATILE: are omitted when 
 submitting a change


==
 OVERVIEW of task #11314:
==

URL:
  <http://savannah.cern.ch/task/?11314>

 Summary: BibEdit: volatile fields should disappear if not
filled in
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-24 09:23
 Should Start On: 2009-09-24 00:00
   Should be Finished on: 2009-09-24 00:00
Category: BibEdit
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


When a record is created fgrom the template and volatile fields are not
filled in, their values should disappear rather than be uploaded as
"VOLATILE: field description"

___

Follow-up Comments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-10-16 09:38      By: Piotr Praczyk 
Solved by the commit: c22035dd9e5314d64593d6360bc9f5dd1ba6ca97

The behaviour of volatile fields has changed a little:

   - Every field whose value starts with VOLATILE: will be 
 displayed as such

   - When displaying a volatile field, the VOLATILE: prefix is
 omitted

   - When editing a volatile field, all the content is 
 automatically selected so that the description appearing by 
 default can be replaced easily
  
   - When editing a normal field, the changes are appended at
 the end rather than replacing (the content is not selected)

   - writing VOLATILE: at the beginning of value changes the 
 field to volatile ( with all the consequences such as not 
 displaying the VOLATILE: prefix and changing the css class

   - all the subfields starting with VOLATILE: are omitted when 
 submitting a change


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-24 09:26  By: Tibor Simko 
See also task #10258.  Kyriakos fixed the removing of empty fields, but
VOLATILE fields were introduced later, so the old removing fix is no longer
fully valid.





___

Carbon-Copy List:

CC Address  | Comment
+-
1576| -COM-
5686| -SUB-




==

This item URL is:
  <http://savannah.cern.ch/task/?11314>

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11595] Enqueue fast modifications in BibEdit

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11595, project CDS Invenio

==
 LATEST MODIFICATIONS of task #11595:
==

Additional Item Attachment, task #11595 (project cdsware):

File name:Size:0 KB

<>

==
 OVERVIEW of task #11595:
==

URL:
  

 Summary: Enqueue fast modifications in BibEdit 
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-10-08 15:13
 Should Start On: 2009-10-08 00:00
   Should be Finished on: 2009-10-08 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


A new mechanism for enqueueing the changes requests. At the moment, if user
makes changes too fast ( faster than the server answers, the record is
assumed modified in a different location )

New changes should be delayed until previous changes are processed
(Another solution would be to extend the model by session identifiers 
more complex)



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-



___

File Attachments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-10-08 15:14  Name: BibEditProblem.png  Size: 13kB   By: piotr
Timestamps problem


==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11595] Enqueue fast modifications in BibEdit

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11595, project CDS Invenio

==
 LATEST MODIFICATIONS of task #11595:
==

Update of task #11595 (project cdsware):

 Assigned to:None => piotr  


==
 OVERVIEW of task #11595:
==

URL:
  

 Summary: Enqueue fast modifications in BibEdit 
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-10-08 15:13
 Should Start On: 2009-10-08 00:00
   Should be Finished on: 2009-10-08 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


A new mechanism for enqueueing the changes requests. At the moment, if user
makes changes too fast ( faster than the server answers, the record is
assumed modified in a different location )

New changes should be delayed until previous changes are processed
(Another solution would be to extend the model by session identifiers 
more complex)



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-



___

File Attachments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-10-08 15:13  Name: BibEditProblem.png  Size: 14kB   By: piotr



==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11595] Enqueue fast modifications in BibEdit

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11595, project CDS Invenio

==
 OVERVIEW of task #11595:
==

URL:
  

 Summary: Enqueue fast modifications in BibEdit 
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-10-08 15:13
 Should Start On: 2009-10-08 00:00
   Should be Finished on: 2009-10-08 00:00
Category: None
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


A new mechanism for enqueueing the changes requests. At the moment, if user
makes changes too fast ( faster than the server answers, the record is
assumed modified in a different location )

New changes should be delayed until previous changes are processed
(Another solution would be to extend the model by session identifiers 
more complex)



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-



___

File Attachments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-10-08 15:13  Name: BibEditProblem.png  Size: 14kB   By: piotr



==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11314] Record templates: volatile fields should disappear if not filled in

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11314, project CDS Invenio

==
 LATEST MODIFICATIONS of task #11314:
==

Update of task #11314 (project cdsware):

 Assigned to:None => piotr  


==
 OVERVIEW of task #11314:
==

URL:
  

 Summary: Record templates: volatile fields should disappear
if not filled in
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-24 09:23
 Should Start On: 2009-09-24 00:00
   Should be Finished on: 2009-09-24 00:00
Category: BibEdit
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


When a record is created fgrom the template and volatile fields are not
filled in, their values should disappear rather than be uploaded as
"VOLATILE: field description"



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #11314] Record templates: volatile fields should disappear if not filled in

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #11314, project CDS Invenio

==
 OVERVIEW of task #11314:
==

URL:
  

 Summary: Record templates: volatile fields should disappear
if not filled in
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-24 09:23
 Should Start On: 2009-09-24 00:00
   Should be Finished on: 2009-09-24 00:00
Category: BibEdit
Priority: 5 - Normal
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


When a record is created fgrom the template and volatile fields are not
filled in, their values should disappear rather than be uploaded as
"VOLATILE: field description"



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[bug #55300] Removing fields/subfields does not work in more complex scenarios

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
bugs #55300, project CDS Invenio

==
 LATEST MODIFICATIONS of bugs #55300:
==

Follow-up Comment #3, bug #55300 (project cdsware):

I mean b1ca733a2653f58e5e79139e816c28b114549240

==
 OVERVIEW of bugs #55300:
==

URL:
  <http://savannah.cern.ch/bugs/?55300>

 Summary: Removing fields/subfields does not work in more
complex scenarios
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-03 17:02
Category: None
Severity: 4 - Important
Priority: 5 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: piotr
Originator Email: 
 Open/Closed: Closed
 Discussion Lock: Any

___


The word undefined appears instead of the field position, preventing some
further code from manipulating on the interface.



___

Follow-up Comments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-17 14:49  By: Piotr Praczyk 
I mean b1ca733a2653f58e5e79139e816c28b114549240

---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-17 14:48  By: Piotr Praczyk 
closed by fdc586f07b38b8a4c079cf0ce63dfc5acfb9f935

---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-04 11:27  By: Piotr Praczyk 
1) After removing a subfield, other subfields of the same field have the word
undefined in the layer name
2) If there are more fields with the same tag, romoving the first and some
subfield of the further does not work at all ! 
(and many similary scenarios)





___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  <http://savannah.cern.ch/bugs/?55300>

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[bug #55300] Removing fields/subfields does not work in more complex scenarios

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
bugs #55300, project CDS Invenio

==
 LATEST MODIFICATIONS of bugs #55300:
==

Update of bug #55300 (project cdsware):

 Open/Closed:Open => Closed 

___

Follow-up Comment #2:

closed by fdc586f07b38b8a4c079cf0ce63dfc5acfb9f935

==
 OVERVIEW of bugs #55300:
==

URL:
  <http://savannah.cern.ch/bugs/?55300>

 Summary: Removing fields/subfields does not work in more
complex scenarios
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-03 17:02
Category: None
Severity: 4 - Important
Priority: 5 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: piotr
Originator Email: 
 Open/Closed: Closed
 Discussion Lock: Any

___


The word undefined appears instead of the field position, preventing some
further code from manipulating on the interface.



___

Follow-up Comments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-17 14:48  By: Piotr Praczyk 
closed by fdc586f07b38b8a4c079cf0ce63dfc5acfb9f935

---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-04 11:27  By: Piotr Praczyk 
1) After removing a subfield, other subfields of the same field have the word
undefined in the layer name
2) If there are more fields with the same tag, romoving the first and some
subfield of the further does not work at all ! 
(and many similary scenarios)





___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  <http://savannah.cern.ch/bugs/?55300>

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[bug #55300] Removing fields/subfields does not work in more complex scenarios

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
bugs #55300, project CDS Invenio

==
 LATEST MODIFICATIONS of bugs #55300:
==

Update of bug #55300 (project cdsware):

 Summary: When removing a subfield from the beginning of the
field in BibEdit, the HTML layer names become incorrect => Removing
fields/subfields does not work in more complex scenarios

___

Follow-up Comment #1:

1) After removing a subfield, other subfields of the same field have the word
undefined in the layer name
2) If there are more fields with the same tag, romoving the first and some
subfield of the further does not work at all ! 
(and many similary scenarios)

==
 OVERVIEW of bugs #55300:
==

URL:
  <http://savannah.cern.ch/bugs/?55300>

 Summary: Removing fields/subfields does not work in more
complex scenarios
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-03 17:02
Category: None
Severity: 4 - Important
Priority: 5 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: piotr
Originator Email: 
 Open/Closed: Open
 Discussion Lock: Any

___


The word undefined appears instead of the field position, preventing some
further code from manipulating on the interface.



___

Follow-up Comments:


---
List-Post: project-invenio-devel@cern.ch
Date: 2009-09-04 11:27  By: Piotr Praczyk 
1) After removing a subfield, other subfields of the same field have the word
undefined in the layer name
2) If there are more fields with the same tag, romoving the first and some
subfield of the further does not work at all ! 
(and many similary scenarios)





___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  <http://savannah.cern.ch/bugs/?55300>

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[bug #55300] When removing a subfield from the beginning of the field in BibEdit, the HTML layer names become incorrect

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
bugs #55300, project CDS Invenio

==
 OVERVIEW of bugs #55300:
==

URL:
  

 Summary: When removing a subfield from the beginning of the
field in BibEdit, the HTML layer names become incorrect
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-09-03 17:02
Category: None
Severity: 4 - Important
Priority: 5 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: piotr
Originator Email: 
 Open/Closed: Open
 Discussion Lock: Any

___


The word undefined appears instead of the field position, preventing some
further code from manipulating on the interface.





___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[task #10267] BibRecord: fix record_delete_subfield() traceback

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
task #10267, project CDS Invenio

==
 LATEST MODIFICATIONS of task #10267:
==

Update of task #10267 (project cdsware):

 Assigned to: bthiell => piotr  


==
 OVERVIEW of task #10267:
==

URL:
  

 Summary: BibRecord: fix record_delete_subfield() traceback
 Project: CDS Invenio
Submitted by: simko
Submitted on: 2009-06-30 12:04
 Should Start On: 2009-06-30 00:00
   Should be Finished on: 2009-06-30 00:00
Category: BibEdit
Priority: 9 - Immediate
  Status: None
 Privacy: Public
Percent Complete: 0%
 Assigned to: piotr
 Open/Closed: Open
 Discussion Lock: Any
  Effort: 0.00

___


Calling record_delete_subfield() tracebacks with:

  File "/usr/lib/python2.5/site-packages/invenio/bibrecord.py", line 510, in
record_delete_subfield
if subfield_code != subfield[0]]

  TypeError: 'tuple' object does not support item assignment

Besides a fix, a test case would be nice to prevent this from happening
again.



___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -UPD-
1576| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/

[bug #54847] Bibupload crasches when wrong input prrovided

[noreply [Piotr Praczyk]]

This is an automated notification sent by LCG Savannah.
It relates to:
bugs #54847, project CDS Invenio

==
 OVERVIEW of bugs #54847:
==

URL:
  

 Summary: Bibupload crasches when wrong input prrovided
 Project: CDS Invenio
Submitted by: piotr
Submitted on: 2009-08-26 11:47
Category: None
Severity: 3 - Normal
Priority: 5 - Normal
  Item Group: None
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email: 
 Open/Closed: Open
 Discussion Lock: Any

___


If a wrong input file is provided, that might be a result of the wrong choice
in the Bibharvest admin interface, bibupload task fails ( a Python error)
instead of writing some meaningful message to the error log file.

Example:

User configures the Harvesting from arXiv. The processing mode is set to
harvest-upload.
The oaiharvest task downloads the recods that are in the OAI format ( not
accepted by the bibupload ).
Bibupload runs and fails without any meaningful message for the user. Only
the intervention of the system administrator having access to the logs can
unveil the reason of the crash. If user does not guess what went wrong,
he(she) is not provided with any meaningful message.



---> maybe an interface allowing to view recent bibupload crashes should be
provided ? (if not present yet)




the Python exception ebtained:

2009-08-26 11:46:04 -> AttributeError: 'NoneType' object has no attribute
'get'
No client information available
Forced traceback (most recent call last):
  File "/opt/cds-invenio/bin/bibupload", line 55, in ?
main()
  File "/usr/lib64/python2.3/site-packages/invenio/bibupload.py", line 1820,
in main
task_run_fnc=task_run_core)
  File "/usr/lib64/python2.3/site-packages/invenio/bibtask.py", line 289, in
task_init
ret = _task_run(task_run_fnc)
Traceback (most recent call last):
  File "/usr/lib64/python2.3/site-packages/invenio/bibtask.py", line 755, in
_task_run
if callable(task_run_fnc) and task_run_fnc():
  File "/usr/lib64/python2.3/site-packages/invenio/bibupload.py", line 1969,
in task_run_core
oai_rec_id = record_id)
  File "/usr/lib64/python2.3/site-packages/invenio/bibupload.py", line 169,
in bibupload
rec_id = retrieve_rec_id(record, opt_mode)
  File "/usr/lib64/python2.3/site-packages/invenio/bibupload.py", line 652,
in retrieve_rec_id
CFG_BIBUPLOAD_EXTERNAL_OAIID_TAG[4:5] != "_" and \
  File "/usr/lib64/python2.3/site-packages/invenio/bibrecord.py", line 213,
in record_get_field_instances
for possible_field_instance in rec.get(tag, []):
AttributeError: 'NoneType' object has no attribute 'get'




___

Carbon-Copy List:

CC Address  | Comment
+-
5686| -SUB-




==

This item URL is:
  

___
  Message sent via/by LCG Savannah
  http://savannah.cern.ch/