pyarrow inputstream snappy

Hi, I'm trying to read snappy file on HDFS using inputstream, but got the error: > > with fs.open_input_stream(read_path, **open_stream_args) as f: > File "pyarrow/_fs.pyx", line 627, in > pyarrow._fs.FileSystem.open_input_stream > File "pyarrow/_fs.pyx", line 557, in >

[Computer-go] [cfp] General Video Game AI Learning Competition @IEEE CoG2021

Apologies for cross-posting *[Call for participants] GVGAI Learning Competition@IEEE CoG2021* Submission deadline: 31 July, 2021 Competition webpage: https://github.com/SUSTechGameAI/GVGAI_GYM Competition framework: http://aingames.cn/gvgai/cog2021/

[jira] [Commented] (HDFS-12571) Ozone: remove spaces from the beginning of the hdfs script

[ https://issues.apache.org/jira/browse/HDFS-12571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125635#comment-17125635 ] Jialin Liu commented on HDFS-12571: --- as of June 14 2020, I'm still seeing the same issue: {code:java

[Computer-go] CFP: WCCI/CEC2020 Special Session on Games - Extended submission deadline: 30 Jan 2020

*Apologies for cross-posting* *CFP of WCCI/CEC2020 Special Session on Games*: http://www.liujialin.tech/events/wcci2020-games.html *Extended submission deadline: 30 Jan 2020* Submission link: https://ieee-cis.org/conferences/cec2020/upload.php Please select S02. Special Session on Games

[Computer-go] CFP: WCCI/CEC2020 Special Session on Games - Submission deadline: 15 Jan 2020

*Apologies for cross-posting* *CFP of WCCI/CEC2020 Special Session on Games*: http://www.liujialin.tech/events/wcci2020-games.html Submission deadline: 15 Jan 2020 Please select CEC-21 Special Session on Games when submitting papers. *Call

[Computer-go] [CFP] Competition and Benchmark Track at 2019 IEEE Conference on Games (former IEEE CIG)

venue for it. *The deadline for short, vision, demo and COMPETITION papers is the 14th of May, 2019. * Have a look, and best regards, and possibly see you in London. Christoph Salge & Jialin Liu Competition and Benchmark Track Chairs The 2019 IEEE Conference on G

[Computer-go] [CFP] Special Session on Games@IEEE CEC2019 (Extended deadline: 31 Jan 2019)

care, education or training) - Results of game-based CI and open competitions *Organizers* *Jialin Liu, liujl(at)sustc.edu.cn <http://sustc.edu.cn/>* Research Assistant Professor, Dept. of Computer Science and Engineering, Southern University of Science and Technology, China *Da

[Computer-go] [CFP] Special Session on Games@IEEE CEC2019 (Extended deadline: 21 Jan 2019)

care, education or training) - Results of game-based CI and open competitions *Organizers* *Jialin Liu, liujl(at)sustc.edu.cn <http://sustc.edu.cn/>* Research Assistant Professor, Dept. of Computer Science and Engineering, Southern University of Science and Technology, China *Da

[jira] [Commented] (SPARK-26261) Spark does not check completeness temporary file

[ https://issues.apache.org/jira/browse/SPARK-26261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710883#comment-16710883 ] Jialin LIu commented on SPARK-26261: Our initial test is: We start a word count workflow including

[jira] [Created] (SPARK-26261) Spark does not check completeness temporary file

Jialin LIu created SPARK-26261: -- Summary: Spark does not check completeness temporary file Key: SPARK-26261 URL: https://issues.apache.org/jira/browse/SPARK-26261 Project: Spark Issue Type

[jira] [Created] (SPARK-26197) Spark master fails to detect driver process pause

Jialin LIu created SPARK-26197: -- Summary: Spark master fails to detect driver process pause Key: SPARK-26197 URL: https://issues.apache.org/jira/browse/SPARK-26197 Project: Spark Issue Type

[Computer-go] [CFP] Special Session on Games@IEEE CEC2019, Wellington, New Zealand (10-13 June 2019)

care, education or training) - Results of game-based CI and open competitions *Organizers* *Jialin Liu, liujl(at)sustc.edu.cn <http://sustc.edu.cn>* Research Assistant Professor, Dept. of Computer Science and Engineering, Southern University of Science and Technology, China *Da

[Openstack] error in deleting objects

Hi, When I tried to delete all objects, with command 'delete -a', it seems some container are not found: Error Deleting: some.h5_9958: u"Container u'some.h5_9958' not found" container:some.h5_9958 Does anyone know how to deal with this? Best, Jialin

Re: [ceph-users] IO to OSD with librados

Thanks for the advice, Dan. I'll try to reconfigure the cluster and see if the performance changes. Best, Jialin On Tue, Jun 19, 2018 at 12:02 AM Dan van der Ster wrote: > On Tue, Jun 19, 2018 at 1:04 AM Jialin Liu wrote: > > > > Hi Dan, Thanks for the follow-ups. > >

Re: [ceph-users] IO to OSD with librados

/monitoring-osd-pg/#monitoring-osds the librados can talk to the OSDs directly, what am I missing here? Best, Jialin NERSC/LBNL > On Mon, Jun 18, 2018 at 3:43 PM Jialin Liu wrote: > > > > Hi, To make the the problem clearer, here is the configuration of the > cluster: >

Re: [ceph-users] IO to OSD with librados

: 48 osds: 48 up, 48 in flags sortbitwise,require_jewel_osds pgmap v117283: 3136 pgs, 11 pools, 25600 MB data, 510 objects 79218 MB used, 1047 TB / 1047 TB avail 3136 active+clean On Mon, Jun 18, 2018 at 1:06 AM Jialin Liu wrote: > Thank you

Re: [ceph-users] IO to OSD with librados

.77 query | jq .up > [ > 58, > 49, > 31 > ] > > Otherwise I answered your questions below... > >> On Sun, Jun 17, 2018 at 8:29 PM Jialin Liu wrote: >> >> Hello, >> >> I have a couple questions regarding the IO on OSD via librados. >> &

Re: [ceph-users] IO to OSD with librados

Sorry about the misused term 'OSS: object storage server' (a term often used in Lustre filesystem), what I meant is 4 hosts, each manages 12 OSDs. Thanks for anyone who may answer any of my questions. Best, Jialin NERSC/LBNL On Sun, Jun 17, 2018 at 11:29 AM Jialin Liu wrote: > Hello, &g

[ceph-users] IO to OSD with librados

Hello, I have a couple questions regarding the IO on OSD via librados. 1. How to check which osd is receiving data? 2. Can the write operation return immediately to the application once the write to the primary OSD is done? or does it return only when the data is replicated twice? (size=3) 3.

Re: [ceph-users] how to build libradosstriper

questions are: 1. Does librados use direct IO? 2. Is there option to leverage page cache? 3. What is the strategy to get the peak bandwidth on a rados object store? 4. Can I use MPI for parallel IO? Best, Jialin NERSC On Wed, May 30, 2018 at 11:08 AM, Jialin Liu wrote: > Thanks Kefu. >

Re: [ceph-users] how to build libradosstriper

Thanks Kefu. Best, Jialin NERSC On Tue, May 29, 2018 at 11:52 PM, kefu chai wrote: > On Wed, May 30, 2018 at 11:53 AM, Jialin Liu wrote: > > Hi Brad, > > > > You are correct. the librados.so has the symbol but what I copied is a > wrong > > file. >

Re: [ceph-users] how to build libradosstriper

any performance number yet. ps. I also found that the stripe unit must be set before object_size properly. Thanks much. Best, Jialin NERSC On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard wrote: > On Wed, May 30, 2018 at 11:52 AM, Jialin Liu wrote: > > Thanks Brad, > > I run nm on

Re: [ceph-users] how to build libradosstriper

000) > /lib64/ld-linux-x86-64.so.2 (0x2aaab000) I also found this thread: http://tracker.ceph.com/issues/14788 which looks similar to the error I run into, and that thread mentioned the version between the headers file and the .so file, I think in my case both of them are v10.2.10, which s

[ceph-users] how to build libradosstriper

Hi, I'm trying to use the libradosstriper api, but having some trouble in linking to lradosstriper. I copied only the `required' libraries from an pre-installed ceph (10.2.10), and put them under my local directory /rados_install/lib and rados_install/include, on a linux machine.

Re: [openstack-dev] container name in swift

e are no character > restrictions. > > [1] https://docs.openstack.org/swift/latest/api/object_api_ > v1_overview.html > > > > 2018-04-02 17:00 GMT-03:00 Jialin Liu <jaln...@lbl.gov>: > >> Hi John, >> What is allowed in container name, but not in object name? &g

Re: [openstack-dev] container name in swift

Hi John, What is allowed in container name, but not in object name? I need a way to distinguish their name.. Best, Jialin On Mon, Apr 2, 2018 at 11:56 AM, John Dickinson <m...@not.mn> wrote: > no > > On 2 Apr 2018, at 11:46, Jialin Liu wrote: > > > Hi, > > Can

[openstack-dev] container name in swift

Hi, Can a container name in openstack swift contains / ? e.g., abc/111/mycontainer Best, Jialin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:

[Computer-go] FDG18 - Call for Competitions

[Apologies for cross posting.] The Foundations of Digital Games 2018 (FDG18) Organizing Committee invites proposals for new competitions. Proposals are due by February 28, 2018, and will be reviewed based on their relevance to the FDG community. The competitions involve well-known games,

Re: [Numpy-discussion] Extending C with Python

With PyObject_CallMethod(pInstance, method_name, "O", py_dims); Can I pass in a reference and modify its content in python? Best, Jialin On Sat, Feb 3, 2018 at 10:29 AM, Jialin Liu <jaln...@lbl.gov> wrote: > Thank you guys. > > Best, > Jialin > > Sent from my

Re: [Numpy-discussion] Extending C with Python

Thank you guys. Best, Jialin Sent from my iPhone > On Feb 3, 2018, at 8:11 AM, Sylvain Corlay wrote: > > You can also check out pybind11, xtensor, and xtensor-python > > The latter enables a high-level numpy-like API on the C++ side. > > You can check out the

Re: [Numpy-discussion] Extending C with Python

Amazing! It works! Thank you Robert. I've been stuck with this many days. Best, Jialin LBNL/NERSC On Tue, Jan 30, 2018 at 10:52 PM, Robert Kern <robert.k...@gmail.com> wrote: > On Wed, Jan 31, 2018 at 3:25 PM, Jialin Liu <jaln...@lbl.gov> wrote: > >> Hello, >&g

[Numpy-discussion] Extending C with Python

Hello, I'm extending C with python (which is opposite way of what people usually do, extending python with C), I'm currently stuck in passing a C array to python layer, could anyone plz advise? I have a C buffer in my C code and want to pass it to a python function. In the C code, I have:

[Computer-go] Final CFP: IEEE Transactions on Games: Special Issue on "Game Competition Frameworks for Research and Education"

[Apologises for multiple postings] Call for Papers - IEEE Transactions on Games (ToG) Special Issue on Game Competition Frameworks for Research and Education Editors: Jialin Liu, Diego Perez-Liebana, Tristan Cazenave, Ruck Thawonmas Submission Deadline: 8 February 2018

[Computer-go] Final CFP: IEEE WCCI/CEC Special Session - Computational Intelligence for Games

[Apologises for multiple postings] Call for Papers: Special Session on Computational Intelligence for Games IEEE WCCI / CEC 2018 <http://www.ecomp.poli.br/~wcci2018/> - Rio de Janeiro, Brazil, 8-13 July 2018 Chairs: Daniel Ashlock, Jialin Liu, Santiago Ontañón Webpages: http://www.ecomp.p

[Computer-go] [Deadline extended] CFP: IEEE WCCI-CEC Special Session - Computational Intelligence for Games

[Apologises for multiple postings] Call for Papers: Special Session on Computational Intelligence for Games IEEE WCCI / CEC 2018 <http://www.ecomp.poli.br/~wcci2018/> - Rio de Janeiro, Brazil, 8-13 July 2018 Chairs: Daniel Ashlock, Jialin Liu, Santiago Ontañón Webpage

[Computer-go] 3rd CFP: IEEE WCCI/CEC Special Session - Computational Intelligence for Games

[Apologises for multiple postings] Call for Papers: Special Session on Computational Intelligence for Games IEEE WCCI / CEC 2018 <http://www.ecomp.poli.br/~wcci2018/> - Rio de Janeiro, Brazil, 8-13 July 2018 Chairs: Daniel Ashlock, Jialin Liu, Santiago Ontañón Webpage

[openstack-dev] Can swift leverage os page buffer?

Hi, I'm new to openstack swift, I'm a HPC user, by several days of exploration of swift, I have some naive questions: 1. Can swift, e.g., PUT, leverage OS' page buffer?; Is there a Linux kernel module for swift? What are the existing optimizations for caching the write in order to gain better

[openstack-dev] leverage page cache in openstack swift

Hi, Is there any existing work that leveraging operating system's page cache for swift? like many other parallel file systems, lustre, the IO is cached in buffer and call is returned immediately to user space. Best, Jialin __

Re: [openstack-dev] how to write python object in openstack swift

://docs.openstack.org/swift/latest/overview_large_ > objects.html#module-swift.common.middleware.slo > > --John > > > > On 5 Oct 2017, at 11:24, Jialin Liu wrote: > > Thank you John, > > I think I figured out the way. My case is a little bit rare as I'm dealing > wi

Re: [openstack-dev] how to write python object in openstack swift

e Swift client (i.e. API user), need to dump those bytes on > the network to send them to Swift. As long as you're transforming your > Python object[s] in some regular way that makes sense in your application, > it doesn't matter what bytes you send to Swift. > > --John > > On

[openstack-dev] how to write python object in openstack swift

Hi, It seems to me that openstack swift only supports file upload/download, is it possible to put a python object to swift store? The doc says we could use file-like object, e.g., StringIO, but this is very limited. I'd like to write a numpy array or other python object into the swift store, can

[Computer-go] 1st CFP: IEEE Transactions on Games (ToG): Special Issue on "Game Competition Frameworks for Research and Education"

[Apologises for multiple postings] Call for Papers IEEE Transactions on Games (ToG) Special Issue on Game Competition Frameworks for Research and Education Editors: Jialin Liu, Diego Perez-Liebana, Tristan Cazenave, Ruck Thawonmas Submission Deadline: 8 January 2018

[UAI] Call for papers: Special Session on Computational Intelligence and Games (CIG) at IEEE CEC2017

hool of Computer Studies, University of Málaga, Span Jialin Liu, jialin@essex.ac.uk School of Computer Science and Electronic Engineering, University of Essex, UK Dr Jialin Liu PhD Senior Research Officer CSEE, University of Essex, UK E jialin@essex.ac.uk ► <http://www.essex.ac.uk/c

Re: [Hdf-forum] h5copy, segmentation fault

Hi, It turns out that the data is highly compressed, once loaded in memory, it consumes 100G or more memory which exceeds the single node RAM capacity. I think that maybe the reason, Thanks. Best, Jialin > On Jul 14, 2016, at 2:09 PM, Jialin Liu <jaln...@lbl.gov> wrote: > > H

[Hdf-forum] h5copy, segmentation fault

Hi, I’m using the h5copy to convert an netcdf file to hdf5, it returns segmentation fault, the file size is 24G, the specific dataset that i tried to convert is H5T_STD_I16LE and ( 1617, 4320, 8640). Any idea about this error? Thanks, Best,

Re: Is storage resources counted during the scheduling

-scheduling.html#scheduling-within-an-application> > > On Mon, Apr 11, 2016 at 3:15 PM, Jialin Liu <jaln...@lbl.gov > <mailto:jaln...@lbl.gov>> wrote: > Hi Spark users/experts, > > I’m wondering how does the Spark scheduler work? > What kind of resources will b

Is storage resources counted during the scheduling

Hi Spark users/experts, I’m wondering how does the Spark scheduler work? What kind of resources will be considered during the scheduling, does it include the disk resources or I/O resources, e.g., number of IO ports. Is network resources considered in that? My understanding is that only CPU

Re: spark launching range is 10 mins

Hi, I have set the partitions as 6000, and requested 100 nodes, with 32 cores each node, and the number of executors is 32 per node spark-submit --master $SPARKURL --executor-cores 32 --driver-memory 20G --executor-memory 80G single-file-test.py And I'm reading a 2.2 TB, the code, just has

[Hdf-forum] unable to use hdf5-java

Hi, I want to use the hdf5 java library on mac, but always got this error: I always got this error: java.lang.UnsatisfiedLinkError: no jhdf5 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864) … The test program I used is H5Ex_D_Hyperslab.java which is downloaded

[Hdf-forum] login password

Hi, I try login onto the h5serv http://tall.data.hdfgroup.org:7253/ but it requires username and password. Where can I have one? Thanks, Jialin___ Hdf-forum is for HDF software users discussion.

a error in git documentation

In http://git-scm.com/book/en/Git-Basics-Recording-Changes-to-the-Repository Notice how you don’t have to run git add on the benchmarks.rb file in this case before you commit. I think how should be changed to now-- To unsubscribe from this list: send the line unsubscribe git in the body of a