Invenio Developer Forum: Status update, brainstorming outcome and tutorials
Dear all, Invenio Developer Forum starts today at 16:00 CET. See https://indico.cern.ch/event/609823/ You can join remotely by Vidyo via the link on the Indico agenda. Agenda includes: - Quick status update on v3 - Outcome of brainstorming session in IT on how to make it easy to contribute to Invenio. - Tutorials coordiation (please join if you go to IUGW) Best regards, Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Developer Forum on Monday
Dear all, Invenio Developer Forum on Monday is dedicated to IUGW planning (+ a quick Invenio v3 status update). https://indico.cern.ch/event/612997/ IUGW is rapidly approaching. Timetable is almost final, we just need to distribute the CERN/Tind contributions on people. Please join if you are going to the IUGW. Cheers, Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - cancelled
Dear all, Today’s Invenio Developer Forum is cancelled. Cheers, Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - 2017-01-30 - Invenio v3 Road Map for Q1
Dear all, The next Invenio Developer Forum will take place on Monday January 30th 2017 at 16:00-16:45 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.html=ygjcGzMEk8re Today, I’ll present the Feb-March road map for Invenio v3. https://indico.cern.ch/event/609749/ Cheers, Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - Invenio User Group Workshop Brainstorming
Dear all, The next Invenio Developer Forum will take place on Monday January 23th 2017 at 16:00 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://indico.cern.ch/event/604648/ https://vidyoportal.cern.ch/flex.html?roomdirect.html=ygjcGzMEk8re Today, we’ll brainstorm the Invenio User Group Workshop programme: See https://indico.cern.ch/event/604648/ for general workshop themes Please come to the meeting today if you plan to join the workshop so we can coordinate presentations. Best regards, Lars Holm Nielsen --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - 2017-01-16 - Persistent Identifier Versioning - Use cases and mock-ups
Dear all, The next Invenio Developer Forum will take place on Monday January 16th 2017 at 16:00 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.html=ygjcGzMEk8re Today, @nharraud @kryzsztof @slint and @lnielsen will use cases and mock-ups for persistent identifier versioning: https://indico.cern.ch/event/604400/ Best regards Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Developer Forum - CANCELLED - Monday January 9th 2017
Dear all, Today's Invenio Developer Forum is cancelled. Best regards, Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - CANCELLED - Monday December 12th 2016
Dear all, Today's Invenio Developer Forum is cancelled. Best regards Lars --- Lars Holm Nielsen CERN, IT Department http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
IDUtils v0.1.0 is released
=== IDUtils v0.1.0 is released === IDUtils v0.1.0 was released on 2015-07-02 About - Small library for persistent identifiers used in scholarly communication. Installation $ pip install idutils Documentation - http://idutils.readthedocs.org/en/v0.1.0 Homepage https://github.com/inveniosoftware/idutils Good luck and thanks for choosing IDUtils. | Invenio Development Team | Email: i...@invenio-software.org | Twitter: http://twitter.com/inveniosoftware | GitHub: http://github.com/inveniosoftware | URL: http://invenio-software.org
Invenio Developer Forum - Killing legacy - Monday June 8th at 16:45 CET
Hello: The next Invenio Developer Forum will take place on Monday June 8th at 16:45 CET 2015 at 16:45 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re and Indico event: https://indico.cern.ch/event/400461/ Today, we will discuss legacy deprecation plan which proactively will be removing all legacy modules even if there wouldn't be any immediately usable Flask-ified counterpart yet. The goal is to aim for PostgreSQL support in 2.2 and Elasticsearch support in 2.3. Best regards Lars --- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - free bazaar - Monday February 9th at 16:45 CET
Hello: The next Invenio Developer Forum will take place on Monday February 9th 2015 at 16:45 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re Today, we shall have a free bazaar session. Best regards Lars --- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 | Twitter: @larshankat | Skype: larsholm-hankat
Invenio Developer Forum - free bazaar - TODAY at 16:45 CEST
Hi, The next Invenio Developer Forum will take place on Monday October 13th 2014 at 16:45 CEST in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re Today, there's a free bazar. Cheers, Lars -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927
Invenio Developer Forum (today 16.45)
Hi, The next Invenio Developer Forum will take place today at 16:45 CEST in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re *Metadata extraction:* Today, Kostas will give a short presentation on ongoing work on metadata extraction from PDFs. *Unresolved Pull requests/RFCs**:* If you have a PR stuck, please notify me, and I'll put it on the agenda for DevForum. Cheers, Lars -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927
Invenio Developer Forum
Hi, The next Invenio Developer Forum will take place today at 16:45 CEST in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at: https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re Today, we thought of discussing the use of the deposition module for anonymous users as well as sign in with orcid/arxiv accounts. Cheers, Lars P.S Let me know if you have any wishes for topic for next weeks meeting. -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927
Re: Invenio v1.1.3 is released
(#1582); bug fix for SPIRES date math search *) WebSession: fix mail cookie expiration (#1596) *) WebSubmit: fix for typo in Shared_Functions; optional pdftk regression tests *) dbquery: closes redundant connection *) git: addition of compile to gitignore; new entry in gitignore *) global: language value always in link URLs *) installation: pip requirement version updates; pip requirements; no user prompt for warnings; empty Travis configuration; location of jquery-1.7.1.min.js; location of flot; information about unidecode; fix autotools rsync instructions *) intbitset: no crash when intbitset is on rhs (#1287) *) inveniocfg: fix for mod_headers *) kwalitee: list comprehensions instead of lambdas; compatibility with pylint 1.0.0 Download: - http://invenio-software.org/download/invenio-1.1.3.tar.gz http://invenio-software.org/download/invenio-1.1.3.tar.gz.md5 http://invenio-software.org/download/invenio-1.1.3.tar.gz.sig Installation notes: --- Please follow the INSTALL file bundled in the distribution tarball. Upgrade notes: -- Please proceed as follows: a) Stop your bibsched queue and your Apache server. b) Install the update: $ tar xvfz invenio-1.1.3.tar.gz $ cd invenio-1.1.3 $ sudo rsync -a /opt/invenio/etc/ /opt/invenio/etc.OLD/ $ sh /opt/invenio/etc/build/config.nice $ make $ make check-upgrade $ sudo -u www-data make install $ sudo rsync -a /opt/invenio/etc.OLD/ \ --exclude invenio-autotools.conf \ /opt/invenio/etc/ # Note: if you are upgrading from previous stable release # series (v0.99 or v1.0), please don't rsync but diff, in order # to inspect changes and adapt your old configuration to the # new v1.1 release series. $ sudo -u www-data /opt/invenio/bin/inveniocfg --update-all $ sudo -u www-data /opt/invenio/bin/inveniocfg --upgrade c) Restart your Apache server and your bibsched queue. d) If your records may contain `oe` UTF-8 graphemes, please schedule reindexing of concerned indexes at a convenient time, for example: $ sudo -u www-data /opt/invenio/bin/bibindex -w title -f10 -R - end of file - -- Surendran K.K. 020-27298293 -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927
PU: Git hooks kwalitee checks...
Hi, *For PU-developers only**:* Every time you submit a pull request to PU there's currently a kwalitee check which checks PEP8, copyright year, pyflakes and the commit message format. All these checks you can now run locally via git hooks: pip install https://github.com/jirikuncar/invenio-kwalitee.git cd ~/src/invenio kwalitee githooks install This will prevent you from committing something that doesn't pass the checks (note, the checks can be disabled pretty easily if you want to ignore the advice ;-) More info is available https://github.com/jirikuncar/invenio-kwalitee Please give it a run, send bug reports, and happily fork away or let us know what to improve by submitting an issue. Lastly, I can't stress enough, that by far the easiest is to plug the checks directly in your editor. Personally I use Sublime 3 with https://github.com/DamnWidget/anaconda, but there's many other alternatives for whatever editor you prefer. Chance is that the developer next to you already have these checks, so just ask if you don't know what to do. Cheers, Lars -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: [pu jsonalchemy] Aggregation of several fields into now
On 27.03.2014 15:38, Esteban Gabancho wrote: Hey guys! I have a question about the aggregation of several fields into one. Taking the example of the authors, lets say I have two fields `_first_author` and `_additional_authors` and I want to aggregate then into `authors`. The common case, and the easiest, is when I have one `_first_author` and cero or more `_additional_authors`, in which case I just put a list with the authors (what else right? :-) The problem, or the question, comes when I don’t have a `_first_author` in which case I’m not sure about the content of the `authors` field, it could be i) only the list of `_additional_authors` or ii) `None` follow by the the list of `_additional_authors`. I think the second solution is the closest one to reality, the `None` express that the record doesn’t have a first author. And I also think that we could apply this solution for other cases where we have this kind of situation (like with the `110__` and `710__`). In Zenodo i'm just interested in the list of authors, and the first in the list is by definition the first author. The first author/additional authors are somehow an artifact of having to store in MARC as master format. I'm not sure what we would be the most appropriate solution. Cheers, Lars What do you think? Lars, as you have already pu in production, how do you deal with this problem? Cheers, -- Esteban J. G. Gabancho -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: [pu-branch] Deposit submission upload
On 17 Mar 2014, at 17:42, Pedro Miguel Paiva Gaudencio
pedro.gauden...@cern.ch wrote:
Hi there,
Add the recid to the sip.metadata (which create_recid() will do for you). If
you jsonalchemy is setup correctly, it should include 001 tag with the recid
in the generated marcxml available in sip.package, and upload_record_sip will
run bibupload -r. This you will usually just run after the user hits submit,
and thus there's not much difference from waiting until bibupload runs,
except making your life easier, by already having the link.
Yeah, probably my jsonalchemy is not doing something, because the recid is
not included in the generated marcxml.
Did you install the invenio-demosite?
No. First generate recid, stick in sip.metadata, then generate marcxml and
stick in sip.package, then bibupload -r. The sip can't and shouldn't be
edited after it's been given to bibupload.
Ok, that's done except for including the recid in the marcxml.
I tried Tibor's approach, but the thing is that bibupload delays the
records...I mean, the recid would be successfully assigned after the record
is created (in theory), but since bibupload/bibsched schedules their
insertion, they stand waiting on queue and when I query the last record
inserted it usually isn't the last submitted one.
Yes, I think you should only use Tibor’s approach if you don’t need the recid
afterwards. Querying for the last recid is an unreliable way to obtain it.
You can test it like this:
from invenio.modules.records.api import Record
r = Record.create({‘recid': 1234}, 'json')
r.produce('json_for_marc’)
This should give you something like this:
[{'005': '20140318071429.0'}, {'001': 1234}]
Cheers,
Lars
Anyway, I think I'll take a look on the jsonalchemy stuff to check why the
recid isn't being included in the marcxml.
Cheers,
Pedro
smime.p7s
Description: S/MIME cryptographic signature
Re: [pu-branch] Deposit submission upload
On 18 Mar 2014, at 17:33, Tibor Simko tibor.si...@cern.ch wrote: On Tue, 18 Mar 2014, Pedro Miguel Paiva Gaudencio wrote: I gave preference to the other approach, seemed less complex and with less post work to do. The major drawback of this approach is vulnerability: if this submission remains open to guests, as most existing INSPIRE submissions are, then watch for script kiddies playing DoS games, or for some automated script going mad, etc. I think both approaches have the same vulnerability? In both cases a recid is generated no matter what - in one case synchronously, in the other case asynchronously. Properly a captcha would be a good idea for INSPIRE submissions? Cheers, Lars Best regards -- Tibor Simko smime.p7s Description: S/MIME cryptographic signature
Re: [pu-branch] Deposit submission taking too long
Hi, Yes, and they also have rate limiting so they will block you if you query too many times. Basically it's just a proof-of-concept, and we might want to take it out of the main Invenio tree until we may have found a working solution. Cheers, Lars On 13.03.2014 11:40, Tibor Simko wrote: On Thu, 13 Mar 2014, Pedro Miguel Paiva Gaudencio wrote: the autocomplete query for sherpa romeo Yes, this used to be really slow last time I looked. A local caching of data would be necessary for any real production. Kostas tried to obtain full local mirror of the database some time ago, but without success. Best regards -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: [pu-branch] Deposit submission upload
Hi Pedro, 1) Generating recid prior to upload: It all depends on the workflow and what you else you need to do. E.g. in Zenodo I need to know the recid prior to uploading, because I use the recid to generate a DOI which goes into the marcxml. Also, knowing the recid before bibupload runs, allows me to quickly generate a preview and record link to the soon to be uploaded record which I can display to the end-user right after they hit submit. In another workflow, it might be fine not to know the recid until after bibupload has been running. 2) JSONAlchemy: All the workflows should be moved to invenio-demosite, where you should have the recid (https://github.com/inveniosoftware/invenio-demosite/blob/pu/invenio_demosite/recordext/fields/atlantis.cfg#L698). It's WIP at the moment, and Esteban should soon have some changes coming in for JSONAlchemy. I.e. you should install invenio-demosite on top of Invenio as well, and we should move the workflows out of Invenio to invenio-demosite. Does that answers your questions? Cheers, Lars On 14.03.2014 17:42, Pedro Miguel Paiva Gaudencio wrote: Hi Lars, I got the deposit submission upload thingy working, just some things left (I think/hope): the marcxml is generated without the 001 (record id - bibupload runs in -r mode in upload_record_sip() and fails because the recid was previously created) and 980(collection information [article, book, preprint, report, etc] - which hides the record by default) fields. I understood that the recid it's not supposed to be present in the new records' marcxml, but if I don't generate the recid (reserved_recid()and create_recid()) the workflow will fail when he gets to run_tasks(). I also understood (not quite sure if I'm right) that when we upload the new deposition, it will be generated a marcxml file from the json that the sip contained. I checked the jsonalchemy.get_producer_rules() and it does not contain any rule for the 'recid', and so this is pobably why it's not being generated (from the json) along with the rest of the xml (on jsonalchemy.wrappers.legacy_export_as_marc()). For the upload of new records to work peacefully we need to: * add the 001 (adding rules for 'recid' in the producer rules?) and 980 fields to the marcxml? * add only the 980 field and always upload_record_sip() in -i mode? Do we need the recid already reserved and created in the sips for the new records before the upload (since when a new record is inserted by bibupload a recid is created for that record)? If so, why? This is my workflow (note that I'm only uploading new records and never editing existing submissions): 1. prefill_draft(draft_id='default'), 2. render_form(draft_id='default'), 3. prepare_sip(), 4. reserved_recid(), 5. create_recid(), 6. process_sip_metadata(process_recjson_new), 7. finalize_record_sip(), 8. upload_record_sip(), 9. run_tasks(update=False) Sorry about the extensive reading. Thanks in advance, Pedro -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Python 2 and 3 compatibility
Dear all, For those that were at the last DevForum we briefly discussed supporting both Python 2 and 3 at the same time, and I promised to send an email around with links. First of all, it will take quite some time before all of Invenio can support Python 3, so this is just an initial kick-off to start thinking about Python 3. If we start already now writing compatible code, we'll have a considerably easier job in the future. I encourage everyone to have a look through the links to see how easy it easy to write Python 2/3 compatible code - it's well worth the investment to upgrade your Python fu now :-) It's after all more than 5 years ago Python 3 was released :-) For all new external modules like Flask-Registry, we should strive to make them Python 3 compatible from the beginning. Writing Python 2/3 compatible source code usually means writing code against Python 2.6, 2.7 and 3.3+. Forget about 2.5- and 3.0/3.1/3.2, since they don't include the necessary syntax changes and features to easily write compatible code. An easy way to get started, is to switch your editor to use Python 3 for syntax error detection, so you'll start seeing the most common errors (e.g. if you're using Sublime 3 that will already be the case). Some of the most easy changes are: Print as a function: print some string - print(some string) Try/except: try: raise Exception() except Exception, e: pass -- try: raise Exception() except Exception *as* e: pass New style classes only: class SomeName - class SomeName(object) Using the six library for (http://pythonhosted.org/six/) compatible code: isinstance(text, basestring) - isinstance(text, six.string_types) I encourage you to look through these links for easy reading on Python 3: http://python3porting.com/differences.html http://python3porting.com/preparing.html http://python3porting.com/noconv.html http://docs.pythonsprints.com/python3_porting/py-porting.html Other good sources of information is: http://python3porting.com/ http://pythonhosted.org/six/ http://docs.python.org/dev/howto/pyporting.html http://docs.python.org/3/whatsnew/index.html The trickiest part of supporting both Python 2 and 3 is definitely the strings, since most of Invenio is dealing with UTF8 byte strings instead of unicode stringss. We'll have to see how we best deal with this. I'm no Python 3 expert myself, so I hope learn from your experiences as well. Best regards, Lars -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
IT Specialist at NASA Astrophysics Data System
Dearl all, To some of you it may be of interest that there's a position open as IT Specialist in NASA ADS (for those who don't know it's like Inspire for Astronomy). Please see the full job description at http://www.cfa.harvard.edu/hr/postings/13-32.html For further information you can contact Alberto Accomazzi aaccoma...@cfa.harvard.edu. Best regards, Lars -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developer Forum - free bazaar
Hello: The next Invenio Developer Forum will take place today at 16:45 CEST in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re. Today we will have free bazaar. Best regards Lars -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developer Forum - free bazaar - right now
Hello: The next Invenio Developer Forum will take place today at 16:45 CEST (now) in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re. Today we shall have the usual free bazaar. Best regards -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Problem caused by socket.setdefaulttimeout()
Hey Jay, Interesting problem. I can reproduce the problem on ZENODO production machines running Python 2.6.6 but I cannot reproduce on my Ubuntu installation with Python 2.7.4 Btw...you might want to look into Celery for distributed processing. Naturally I don't know if it would fit your use-case, but at least I've found celery much less headache than trying to use multiprocessing :-) Best regards, Lars On 08.08.2013 22:31, Luker, James wrote: Hi gang, Does this stacktrace look familiar-ish to anyone? Traceback (most recent call last): File stdin, line 1, in module File /home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/managers.py, line 667, in temp token, exp = self._create(typeid, *args, **kwds) File /home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/managers.py, line 565, in _create conn = self._Client(self._address, authkey=self._authkey) File /home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/connection.py, line 175, in Client answer_challenge(c, authkey) File /home/jluker/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/multiprocessing/connection.py, line 413, in answer_challenge message = connection.recv_bytes(256) # reject large message IOError: [Errno 11] Resource temporarily unavailable We started seeing exceptions like that yesterday when we tried incorporating a new blueprint into our ADS 2.0 Flask app that uses some components of the python mulitprocessing library. Googling turned up a few clues and mentions about socket timeout stuff. I suspected something we were importing was maybe monkey-patching or otherwise messing with sockets, so I painstakingly went through all of our import chains and tracked it back to bibdocfile.py*, where it sets socket.setdefaulttimeout(40). Apparently that's enough to totally break multiprocessing. I don't really understand why, but you can duplicate the error very easily... import socket socket.setdefaulttimeout(40) from multiprocessing import Manager m = Manager() res = m.list([]) Traceback ... I was able to work around the exceptions by simply calling, socket.setdefaulttimeout(None), immediately after the from invenio import ... statements, but I figured I should give you a heads up. Also, Roman mentioned that you'd had problems making use of the multiprocessing library in the past. Surely it's not that simple though. --jay * our app imports some functions like run_sql from invenio -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developer Forum - free bazar - today at 16:45 CEST
Hello: The next Invenio Developer Forum will take place today at 16:45 CEST in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re. There's no topic for today. Best regards -- Lars Holm Nielsen CERN, IT Department, Collaboration Information Services http://zenodo.org | Tel: +41 22 76 79182 | Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
RFC: signalutils
Dear all, Please find following RFCof the upcoming signals integration in Invenio next: http://invenio-software.org/wiki/Tools/Signals Best regards, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Jailbreaking the PDF: a Hackathon
Perhaps of interest: http://scholrev.org/hackathon/ (see under tools) -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Zenodo - Sharing Research Data across Europe - Making Science More Visible
Dear all, ZENODO(http://zenodo.org) just went live! I've appended the press release (http://www.openaire.eu/en/component/content/article/9-news-events/457-zenodo-is-launched) Best regards, Lars *Zenodo - Sharing Research Data across Europe - Making Science More Visible * Newly launched, Zenodo www.zenodo.org offers a one-stop-store for research output. Created by OpenAIRE and CERN, and supported by the European Commission, this new-generation online repository offers its service from the OpenAIRE pan-European initiative, which expands the linking of research output to datasets and funding information, in European and national contexts. **Enabling everyone to Share and Cite Data** Zenodo welcomes multi-disciplinary research data from any individual, scientific community or research institution. Upload allowance is generous (1GB) and can be used by institutions without their own data repository. Based on the same concept as OpenAIRE, which gathers Open Access publications across a variety of funding schemes, Zenodo provides a rich interface to link objects together with funding information. **Supporting the long-tail of research output** Any data uploaded, or collections created are harvestable via OAI-PMH by third parties: expose your collection to PubMedCentral or your local institution. For research institutions who don't want the overhead of establishing their own data repository to support their researchers' scientific output, this is a convenient solution. The repository accepts any data without an obvious service at hand, in a variety of formats. Zenodo fully encourages deposition under an open licence, and while it will also accept other licence types, the Zenodo community will take a lead in signalling the benefits of open licenses such as visibility and credit. **Building Collections for Scientific Communities** Zenodo adds value in that it enables users to have ownership over their unique community collections. For example, an EC funded project might like to create a collaborative space for all its research output, and can assign a range of licenses, including Creative Commons, and each dataset and publication is assigned a DOI. Chris Erdmann, Head Librarian at the Harvard-Smithsonian Center for Astrophysics, says, This will be of great benefit to the global research community. Institutions, together with scholarly communities, are looking for flexible deposit solutions that allow the living scholarly record to be easily curated, exchanged and cited. For the research community, to have a trustworthy publication and sharing mechanism for their scholarly activities at their fingertips, will be hugely beneficial. Brian Hole, CEO at Ubiquity Press says Zenodo is a welcome addition to the options we provide our authors for publishing their data alongside their research articles and data papers. I particularly like the innovative way in which the upload system has been designed to be quick and simple, which directly addresses one of researchers chief complaints about data archiving - that it is time consuming. We will be happy to suggest our authors deposit their underlying datasets at Zenodo. Florida Estrella, Deputy Director of the European Middleware Initiative (EMI) based at CERN, adds Science has entered the age of open. EMI connects scientists and will be able to employ Zenodo's services in a transparent and reliable way. **An easy-to-use workflow** Sign up now for an account at Zenodo and submit your research in easy steps (e.g. via Dropbox). -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Access rights for files
Hi, Unfortunately I don't know of any librarian accessible interfaces to properly set the file. In Orphan repository I have a bibtasklet running in the background which based on what's in the MARC, will ensure that the files has proper access right. I can share if you are interested, but I'm not sure it applies in your specific case. Cheers, Lars On 26/04/13 14:03, Alexander Wagner wrote: Hi! Upon upload, I can easily set up permissions of a file by means of the fft-tags according to our needs. However, till now I did not find a way to change the rights of a full text file on a higher level than an fft tag in marc xml. Or in other words: where is the interface to adopt access rights that could be accessible for our library staff? It seems that managedocfile does not do the trick. I can revise the file therem but the only sensible option for access is public. However, sensible might clash with jurisdiction (as usual ;) This dialogue offers restricted as well, but I have no idea what restricited actually sets. Nor do I see a away to define which restrictions should be applied. (Not even if I allow free form firerule writing, which would probably be a bit technical in some cases as well.) A common use case arises here due to embargoed open access. We get the file for OA from our authors, but it has an embargo for say 12 months. Afterwards, we can enable access to the file. Easy to accomplish, by use of firerules in principle. But: to get the content in the first place and for some other (legal) reasons within websubmit we set the access permissions for the files quite tightly. This allows us to tell our users: upload the file in case of doubt we check anyway. Restrictions here are only staff and the uploader can see the file itself. Then the library checks the submission and if the file in question can be released to the world our librarian would like to set permissions properly. Probably we would like to set something like deny until 2014-01-01. But at the moment I see no way how we can do that if the final permission is not free for world. Thanks for your help! :) Currently still invenio 1.0.2x -- Kind regards, Alexander Wagner Subject Specialist Central Library 52425 Juelich mail : a.wag...@fz-juelich.de phone: +49 2461 61-1586 Fax : +49 2461 61-6103 www.fz-juelich.de/zb/DE/zb-fi Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Access rights for files
The bibtasklet is here http://invenio-software.org/repo/openaire/tree/bibsched/lib/bibsched_tasklets/bst_openaire_check_rights.py (it's in OpenAIRE repository, so it will not work out of the box on standard Invenio. The access rights are very simple, hence the possiblity to do access rights in MARC: open, closed/restricted, embargoed stored in 542_l and embargo date stored in 942__a ( http://openaire.cern.ch/record/5956/export/hm?ln=en) Cheers, Lars On 26/04/13 14:34, Alexander Wagner wrote: On 26.04.2013 14:09, Lars Holm Nielsen wrote: Hi! Unfortunately I don't know of any librarian accessible interfaces to properly set the file. In Orphan repository I have a bibtasklet running in the background which based on what's in the MARC, will ensure that the files has proper access right. I can share if you are interested, but I'm not sure it applies in your specific case. This would be definitely of interest. How do you store the permissions in marc? IMHO even FFT should probably be exposed to bibedit anyway like a usual MARC field. (My librarians can usually easily handle everything that is in Marc but python backend is another story.) -- Kind regards, Alexander Wagner Subject Specialist Central Library 52425 Juelich mail : a.wag...@fz-juelich.de phone: +49 2461 61-1586 Fax : +49 2461 61-6103 www.fz-juelich.de/zb/DE/zb-fi Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Search python code for imports
Hi, I created a small script that will allow you to search for specific imports in your Python source code, useful when you aredoing refactoring. The difference froma grep is that it actuallyparses the Python codeand only take imports into account, so it's more precise. https://pypi.python.org/pypi/py-import-search/ https://pypi.python.org/pypi/py-import-search/0.1 Perhaps it's you'll find it useful as well. Cheers, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Anyone need running BibClassify as standalone (i.e. without all Invenio)?
Dear all, Does anyone need to run bibclassify without having all of Invenio? I'm about to do a bit of cleaning. Certain parts of Invenio messes with sys.path causing some troubles for me :-) One part is bibclassify_config which apparently does it to be able to run outside of an Invenio installation. Is anyone still in need of this? The problem with messing with the sys.path is that you can do both e.g. import invenio.config and import config. The latter version should not be possible since it's outside the package hierarchy and causes conflickts when you have a system-level package with the same name - e.g. invenio.celery and main celery package. More info: http://invenio-software.org/ticket/1459#ticket Cheers, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Overlay example
Hi, I just added a created a quick skeleton for an overlay, with some example files for how to customize Invenio: https://github.com/Zenodo/invenio-overlay-example I created it since it was the second time I needed to help someone to setup an overlay, so I thought it's easier just to have a pre-made skeleton to start from :-) Let me know if there's some examples missing things to change etc. Currently, it's for next, but only because it adds two files which in principle would not interfere with master. There's some info in README and INSTALLATION as well. Cheers, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developer Forum - free bazar - today at 16:45 CET
Hi, The next Invenio Developer Forum will take place today at 16:45 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re. Today, there's a free bazar. Best regards, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: TEI XML import / export
On 22/03/13 13:22, Vít Tuček wrote: Thanks a lot! On 22 March 2013 11:56, Lars Holm Nielsen lars.holm.niel...@cern.ch wrote: Dear Vit, On 22/03/13 08:24, Vít Tuček wrote: Hello everyone, my employer decided to try to move his digital library to invenio and I was tasked to facilitate that process. I kindly ask for your help in assessing the work required to do so. Our data are stored in TEI P5 XML and we already have XSL transforms to extract MARC XML. We were thinking of the following workflow. (Of course any ideas how to proceed in way better suited for Inveio are appreciated.) IMPORT: A daemon checks an assigned directory and if there is a zip file it unzips it and runs the XSL transform on the resulting TEI XML to produce the MARC XML. The MARC XML is then checked for sanity, imported into the library and associated with the original TEI XML. If you already have a MARC XML transformation, then your are nearly done :-) There are several ways to get content into Invenio, and I think which way you use depends mostly on how you want to do the migration. The central part of Invenio responsible for uploading metadata is BibUpload which takes a MARC XML file by default. Additionally we have BatchUploader which is basically a wrapper around BibUpload, which will monitor a directory and import MARC XML files. More details are available here http://invenio-demo.cern.ch/help/admin/bibupload-admin-guide I'm glad to hear that. EXPORT: We would like to be able to export the TEI XML (or it's XSL transforms) as collections from the web interface. For this you would use BibFormat. You would add a new output format ( http://invenio-demo.cern.ch/help/admin/bibformat-admin-guide#addOutputFormat): Give it a Code and content-type (note, the code is important, there's special handling depending on the first letter, especially for x and h). Then add a XSL format template for the output format that transforms MARC XML to TEI XML: http://invenio-demo.cern.ch/help/admin/bibformat-admin-guide#xslFormatTemplate Here's an example of Dublin Core transformation: Output format: http://invenio-software.org/repo/invenio/tree/modules/bibformat/etc/output_formats/XD.bfo?h=maint-1.1 Format template: http://invenio-software.org/repo/invenio/tree/modules/bibformat/etc/format_templates/OAI_DC.xsl?h=maint-1.1 The files live in etc/bibformat/format_templates and etc/bibformat/output_formats Once you have the transformation, each record can be exported individually, or collective from the search page, as well as background job. I am not sure we are on the same page here. Let me try to explain myself better. The TEI XML contains much more than bibliographic data and we would like to be able to store that in Invenio and apply some XSL transforms to it during export. Think of handling a PDF with metadata extraction during import and pdf2html during eport. I think we might be misunderstanding each other with what we mean with import/export. Import: How to get you metadata and files into Invenio. Export: Once the metadata and files are already Invenio, how can you get it out again in another format. For the import, it's all done through BibUpload and a MARCXML file. The MARCXML file can point to files that Invenio also needs to store (e.g. PDF documents, video images etc - see http://invenio-demo.cern.ch/help/admin/bibupload-admin-guide#3.6). This way you get metadata and files into Invenio. There's no restrictions on which files you can associate. Once the metadata and files are stored in Invenio, invenio has some other tools to extract references from PDFs, create thumbnails, classify documents etc. All of them however, just read information which is in inside Invenio (files or metadata), and creates a new MARCXML file (with possible links to files - see FFT tag in link above) and send it to BibUpload. As far as I understand, you are mainly interested in getting data into Invenio, but not out again in different format (ie export). Does this answer your questions? Cheers, Lars Best regards, Lars I installed Invenio locally and took a quick look around. I'm a little bit lost in the documentation right now, but it seems that the import is mostly a matter of proper configuration. As for export, so far it seems to me that some scripting is needed. Best regards, Vit -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: TEI XML import / export
Dear Vit, On 22/03/13 08:24, Vít Tuček wrote: Hello everyone, my employer decided to try to move his digital library to invenio and I was tasked to facilitate that process. I kindly ask for your help in assessing the work required to do so. Our data are stored in TEI P5 XML and we already have XSL transforms to extract MARC XML. We were thinking of the following workflow. (Of course any ideas how to proceed in way better suited for Inveio are appreciated.) IMPORT: A daemon checks an assigned directory and if there is a zip file it unzips it and runs the XSL transform on the resulting TEI XML to produce the MARC XML. The MARC XML is then checked for sanity, imported into the library and associated with the original TEI XML. If you already have a MARC XML transformation, then your are nearly done :-) There are several ways to get content into Invenio, and I think which way you use depends mostly on how you want to do the migration. The central part of Invenio responsible for uploading metadata is BibUpload which takes a MARC XML file by default. Additionally we have BatchUploader which is basically a wrapper around BibUpload, which will monitor a directory and import MARC XML files. More details are available here http://invenio-demo.cern.ch/help/admin/bibupload-admin-guide EXPORT: We would like to be able to export the TEI XML (or it's XSL transforms) as collections from the web interface. For this you would use BibFormat. You would add a new output format ( http://invenio-demo.cern.ch/help/admin/bibformat-admin-guide#addOutputFormat): Give it a Code and content-type (note, the code is important, there's special handling depending on the first letter, especially for x and h). Then add a XSL format template for the output format that transforms MARC XML to TEI XML: http://invenio-demo.cern.ch/help/admin/bibformat-admin-guide#xslFormatTemplate Here's an example of Dublin Core transformation: Output format: http://invenio-software.org/repo/invenio/tree/modules/bibformat/etc/output_formats/XD.bfo?h=maint-1.1 Format template: http://invenio-software.org/repo/invenio/tree/modules/bibformat/etc/format_templates/OAI_DC.xsl?h=maint-1.1 The files live in etc/bibformat/format_templates and etc/bibformat/output_formats Once you have the transformation, each record can be exported individually, or collective from the search page, as well as background job. Best regards, Lars I installed Invenio locally and took a quick look around. I'm a little bit lost in the documentation right now, but it seems that the import is mostly a matter of proper configuration. As for export, so far it seems to me that some scripting is needed. Best regards, Vit -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Celery integration for next
Hi Ferran, I think there should be no problem in your case. On your host, you would install one RabbitMQ server (the broker) - on the broker you would create 1 RabbitMQ virtual host per Apache virtual host. For each invenio installation you would start 1 worker. I know Invenio already have lots of dependencies, but the hope is that this one is worth-while and easy Basically installing RabbitMQ is something like: # Install sudo apt-get install rabbitmq-server # Add user and vhost sudo rabbitmqctl add_user myuser mypassword sudo rabbitmqctl add_vhost myvhost sudo rabbitmqctl set_permissions -p myvhost myuser .* .* .* # Restart sudo service rabbitmq-server restart and starting a Celery worker is just: celery worker -A invenio -B Do you install each Invenio installation in virtual environments? If not, this might be the only issue, however I think at most a worker-start-script per invenio installation would need to be created. Alternatively, we are also thinking of a lite-solution, so you won't even need to install a broker (RabbitMQ) and start the Celery workers. Celery has a flag so that it can run tasks synchronously instead of asynchronously (so the lite version would seem slower, but still do the job in the end). Currently there's an overlap between bibsched and Celery, which we haven't completely sorted out what goes where. For now, bibsched is still the master of bibupload and friends. In the short term it seems most natural that Celery would take over bibtasklets + new territories. On the long run, we'll have to get some experiences first. Cheers, Lars On 20/03/13 08:48, Ferran Jorba wrote: Hello Lars, I've finished initial integration of Invenio in Celery for next: http://invenio-software.org/repo/personal/invenio-lnielsen/commit/?h=next-celeryid=6d09ef545f03edfa6d7c77cd3a2447873b16c87e It basically follows what we discussed in DevForum (https://invenio-software.org/wiki/Tools/Celery/InvenioIntegration). Take a look if you have a minute and let me know if there's issues. [...] Yes, please, I have a doubt: at UAB, we have more than one Invenio installation in the same host, installed as plain users (not root nor www-data), and served by Apache (specifically, apache-itk) with virtual hosts. Will this celery integration be compatible with our setup? Thanks, Ferran PS And congratulations for your zenodo branch. It looks gorgeous! We are constantly looking at it for inspiration, and we are taking some ideas for our forthcoming 1.1 upgrade. -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Celery integration for next
On 20/03/13 09:26, Ferran Jorba wrote: Hello Lars, I think there should be no problem in your case. On your host, you would install one RabbitMQ server (the broker) - on the broker you would create 1 RabbitMQ virtual host per Apache virtual host. For each invenio installation you would start 1 worker. Great, good to know it's so simple! [...] Do you install each Invenio installation in virtual environments? If not, this might be the only issue, however I think at most a worker-start-script per invenio installation would need to be created. No, we are not using virtual environments; I'm trying to keep it as flat as possible, and I'm not using any extra layer that I don't really need. Oki, what we will have to do then is to install a start worker script into path to some invenio/bin which will ensure the right worker is started. Alternatively, we are also thinking of a lite-solution, so you won't even need to install a broker (RabbitMQ) and start the Celery workers. Celery has a flag so that it can run tasks synchronously instead of asynchronously (so the lite version would seem slower, but still do the job in the end). In which situations it «seems slower»? The end user front-end or the librarians-systems back-end? Anywhere a Celery task would be called - front-end/back-end. However, a Celery task would normally only be used when you have any request that takes a long time. One example, would be an export of say 2000 records in format X. With a celery task, we can give back a instant reply to the user, saying we are working on it, and will send him an email once the result is ready (like if you download a folder from Google Docs). With a synchronous task, the end-user will have to wait for a reply from the server until the export is done. Another example could be WebDeposit - the user uploads a file, and a Celery task extracts metadata from the PDF, in the background, while the user can start entering information in the deposit form. WIth the synchronous task, the user will have to wait. So definitely for most cases you do want to run celery, but for small installations without big requirements this can be an easy way to get up and running. A distributed task queue, is one of the most effective ways of speeding up responsiveness of the application. Simply do everything in the background which is necessary for the system to work, but which the user doesn't have to wait to finish. Cheers, Lars Currently there's an overlap between bibsched and Celery, which we haven't completely sorted out what goes where. For now, bibsched is still the master of bibupload and friends. In the short term it seems most natural that Celery would take over bibtasklets + new territories. On the long run, we'll have to get some experiences first. Sure. Thanks for answering so fast, Ferran Cheers, Lars On 20/03/13 08:48, Ferran Jorba wrote: Hello Lars, I've finished initial integration of Invenio in Celery for next: http://invenio-software.org/repo/personal/invenio-lnielsen/commit/?h=next-celeryid=6d09ef545f03edfa6d7c77cd3a2447873b16c87e It basically follows what we discussed in DevForum (https://invenio-software.org/wiki/Tools/Celery/InvenioIntegration). Take a look if you have a minute and let me know if there's issues. [...] Yes, please, I have a doubt: at UAB, we have more than one Invenio installation in the same host, installed as plain users (not root nor www-data), and served by Apache (specifically, apache-itk) with virtual hosts. Will this celery integration be compatible with our setup? Thanks, Ferran PS And congratulations for your zenodo branch. It looks gorgeous! We are constantly looking at it for inspiration, and we are taking some ideas for our forthcoming 1.1 upgrade. -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Celery integration for next
Hi, I've finished initial integration of Invenio in Celery for next: http://invenio-software.org/repo/personal/invenio-lnielsen/commit/?h=next-celeryid=6d09ef545f03edfa6d7c77cd3a2447873b16c87e It basically follows what we discussed in DevForum ( https://invenio-software.org/wiki/Tools/Celery/InvenioIntegration). Take a look if you have a minute and let me know if there's issues. Currently this is the barebones integration - more user documentation is needed eg, and easy management of the workers I haven't looked into. Part of the configuration is taken from invenio.config (installation specific), rest is defined in miscutil/lib/celeryconfig.py (invenio specific). The integration consists of a special Celery loader for Invenio (e.g. Django and the other frameworks also have special loaders), which takes care of loading *_tasks.py files as well as initializing Flask application/request context so that you can use SQLAlchemy modes etc. in your tasks, and that's basically it. *bibsomemodule_tasks.py**:* # Flask _application context_ available here: e.g. import SQLAlchemy models will work. from invenio.celery import celery @celery.task(ignore_result=True) def mytask(x): # do something (flask test _request context_ available here - 1 test request context per task) *othermodule.py:* from invenio.bibsomemodule_tasks import mytask mytask.delay(1) *Starting Celery worker:* celery worker --app=invenio ... Things missing: - Define a nice task example with everything included. - Management: Easy start/stop of workers (define log dirs, pid files etc, default arguments), integration of monitoring etc. - Backport to master: only requires to remove code from the loader that takes care of initializing flask. - How to easily test tasks :-) - FIx a nasty import issues. Cheers, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developer Forum - today at 16:45 CET (i.e in 15 min)
Hello: The next Invenio Developer Forum will take place today at 16:45 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re. Today, is the usual free bazaar. Best regards Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developer Forum - free bazaar - today at 16:45 CET
Hello: The next Invenio Developer Forum will take place today at 16:45 CET in (i) CERN room 31-S-023 and in (ii) Vidyo videoconferencing room at https://vidyoportal.cern.ch/flex.html?roomdirect.htmlkey=ygjcGzMEk8re. Today, we have the usual free bazaar. If nobody have special topics to discuss, I'll give an ultra short demo of http://zenodo-dev.cern.ch (work in progress) and how the new next branch have been customized. Best regards, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Add all search results to a basket?
Hi, Would it be all records for an entire search or just all records displayed on one result page? In case of the latter, there's a toggle all button in the next-branch (see example on: http://invenio-demo-next.cern.ch/search?p=action_search=). Cheers, Lars On 30/11/12 09:07, Ferran Jorba wrote: Hello Alexander, Alexander Wagner a.wag...@fz-juelich.de wrote: Hi! Is there currently a way to add all records retrieved by a search to a basket? I mean without hooking on every hit and then add per page? Probably even some hook all records on this page function? I just got this question from our users and at the moment have to admit that the only answer I know would be to do it per record. (Note: I do not want to store the query in the basket but really the resulting records.) +1 Ferran -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Re: Upgrader script
Hi Piotr,
I'm at French course today from 11-13, so I can stop by your office before or
after.
On 15 Nov 2012, at 21:40, Piotr Praczyk piotr.prac...@cern.ch wrote:
Just a small amendment to the previous e-mail
I have just realised that in the 2nd question, the logical solution would be
in every commit consider present scripts as applied... which basically means
that inveniocfg --create-tables should always mark all upgrade scripts as
applied (this seems not to happen)
I misunderstand something or this is a bug ?
No, you're right. There's an currently an inconsistency between tabcreate.sql,
inveniocfg --create-table and --upgrade. I'll try to see if I can get a fix in
today. The tables created in the upgrader scripts, are currently also created
in tabcreate.sql, so the correct would be to have them marked as alreday
applied.
Piotr
From: Piotr Praczyk
Sent: 15 November 2012 21:37
To: project-invenio-devel (Invenio developers mailing-list);
inspire-...@slac.stanford.edu
Subject: Upgrader script
Hello
I have several questions concerning the Upgrader mechanism introduced to
Invenio.
First of all, could someone who knows the topic better, review my upgrate
module ? It can be seen under:
http://invenio-software.org/repo/personal/invenio-ppraczyk/commit/?h=bibdocfile-squashed-ultimate-rebasedid=be09d38416fe5adc0898f40e94fe80118dedd70e
from invenio.config import CFG_LOGDIR, CFG_SITE_SUPPORT_EMAIL,
CFG_BIBDOCFILE_FILEDIR, CFG_BIBDOCFILE_FILESYSTEM_BIBDOC_GROUP_LIMIT
+from invenio.bibdocfile import BibRecDocs, InvenioBibDocFileError
import os, sys
I guess they are related to your question below, but in the commit they are not
used, and should thus not be imported.
from invenio import bibdocfile
Any non-safe import, should be imported in one of the methods instead of in the
module. In the method you can catch an possible import errors as well, and
raise a RuntimeError. This is to ensure that the module can be loaded even
prior to Invenio being installed (when running make check-upgrade). In the
do_upgrade you can try to import it, catch possible ImportError and bail out
via RuntimeError if it fails.
Logging:
logger = logging.getLogger('invenio_upgrader')
logger.info
For longer running upgrade, using info is fine. For shorter upgrade, don't
bother about info. For your usage, using the logger is fine, but only the
logger.info(). Instead of logger.error you should use raise RuntimeErrorr(msg1,
msg2, ..). Same goes for warnings. Use warnings.warn(msg) instead of
logger.warn(). This will ensure consistency in that error should stop the
migration process, warnings, you are logged and can be continued from. E.g. if
you
I.e:
+logger.error(Failed fixing the record %s % (str(recid)))
Should be raise RuntimeError, or warnings.warn, depending on if you want to
stop the migration script or not. I assume you would want to use warnings.warn
pre_upgrade():
If you raise the runtime error, you'll stop the entire upgrade process, which I
dont' think you want to do. YOu just want to skip it this upgrade right? So
instead, just check if more_info is there, set a global varialbe or similar,
that do_upgrade will check to see it it shoudl skip the upgrade. In the
pre_upgrade, you can issue a warning, saying that it will skip it.
post_upgrade():
If you don't use it, you can just remove the method.
The questions (about what I don't understand after reading the documentation
in inveniocfg_upgrader.py) :
1) Modules are supposed to be standalone, not depending too much on imports
from Invenio unless it is absolutely necessary.
- which imports can be considered necessary ? Upgrade scripts cannot be
ammended in the future (according to the doc), making this work seemingly an
impossible task.
What is meant with cannot be changed in the future, is that if you create a
table, and see that you forgot to add a column, then you should create another
upgrade if it's already committed to master. If there's a bug in the upgrade,
you can naturally fix it.
For instance (example from this upgrade) if I need to import some
configuration variables which have got renamed in one of the commits (in this
example together with moving bibdocfile from websubmit to bibdocfile module).
If we decide to further rename those varaibles, we have to ammend all
upgrade scripts, otherwise it will clearly fail.
Is there a list of safe imports ?
Safe imports are currently run_sql and wait_for_user. When say 10 upgrades are
committed, we'll have a look to see if we need more (very likely). So again,
you can import other mehtods, but if you do, put the import in one of your
methods (e.g. do_upgrade)
2) How can I mark that a certain commit introduces given changes ?
Afetr I have installed the commit and created database using invenioconfig
script, intuitively I would expect
Invenio Developers Forum - today at 16:30 CEST - Bldg 31-S-023
Hi, The next Invenio Developers Forum will take place today starting at 16:30 CEST in the EVO videoconferencing meeting called `Invenio Developers Forum' and in person at CERN in room 31-S-023. There is no particular topic for today, we'll have the usual free bazaar[*]. If anybody is interested I'll demo the new migration/upgrade feature. [*] As discussed in the past, VC screen and sound may remain inactivated unless there is a topic of common interest to everybody present. This is so that various `breakout discussion groups' may happen in parallel. Therefore, if you connect via EVO and notice people being logged in but you cannot see or hear anyone, just say hi and someone will show up. Cheers, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Invenio Developers Forum - NOW - Bldg 31-S-023
Hi, The next Invenio Developers Forum will take place today starting at 16:30 CEST in the EVO videoconferencing meeting called `Invenio Developers Forum' and in person at CERN in room 31-S-023. There is no particular topic for today, we'll have the usual free bazaar[*]. If anybody is interested I'll demo the new migration/upgrade feature. [*] As discussed in the past, VC screen and sound may remain inactivated unless there is a topic of common interest to everybody present. This is so that various `breakout discussion groups' may happen in parallel. Therefore, if you connect via EVO and notice people being logged in but you cannot see or hear anyone, just say hi and someone will show up. Cheers, Lars -- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 Cel: +41 76 672 8927 smime.p7s Description: S/MIME Cryptographic Signature
Reminder (in 20 minutes): Invenio Developers Forum - today at 16:30 CEST - Bldg 31-S-023
Begin forwarded message: From: Lars Holm Nielsen lars.holm.niel...@cern.ch Subject: Invenio Developers Forum - today at 16:30 CEST - Bldg 31-S-023 Date: 30 July 2012 09:54:52 GMT+02:00 To: project-invenio-devel (Invenio developers mailing-list) project-invenio-devel@cern.ch Hello: The next Invenio Developers Forum will take place today starting at 16:30 CEST in the EVO videoconferencing meeting called `Invenio Developers Forum' and in person at CERN in room 31-S-023. If you have tried or seen anything cool (e.g. the python module which just saved your life, or you heard about a cool new product), come and share it. If you have any questions, doubts, wishes, comments, on git, testing, installation, development of Invenio modules such as WebSubmit, BibHarvest, BibUpload, etc., come and share :-) Cheers! Lars --- Lars Holm Nielsen Software Engineer CERN, IT Department, Digital Library Technology Section Office 513/1-014 Tel: +41 22 76 79182 smime.p7s Description: S/MIME cryptographic signature
Re: Invenio HTTP Server with automatic code reloading
On 13 Jun 2012, at 16:28, Tibor Simko wrote: On Thu, 05 Apr 2012, Alessio Deiana wrote: I have been working on a web server that makes it easier to work on invenio. I have been using it for the past month and has been very useful to me. It has quite a few handy features: * auto install modified files and reloads itself to take into account new code * display of errors on the console You can find the project and the source code at this url: https://bitbucket.org/osso/invenio-devserver/overview Can you perhaps amend the configuration part so that it could be shared with my invenio-devscripts: https://github.com/tiborsimko/invenio-devscripts via environment variables? The motivation being that if a developer uses both of these extension tools, then he/she can set up only one set of CFG_INVENIO_PREFIX variables and friends, and it would be recognised everywhere. The configuration part will try to use both VIRTUAL_ENV or CFG_INVENIO_PREFIX to determine the install path, and use CFG_INVENIO_SRC or if not found ~/src/invenio/ for the source path. Cheers, Lars P.S. Have not checked invenio-devserver yet, as I'm using Emacs shortcuts everywhere for reloading. :) Best regards -- Tibor Simko smime.p7s Description: S/MIME cryptographic signature
