pyarrow inputstream snappy

[Jialin Liu]

Hi, I'm trying to read snappy file on HDFS using inputstream, but got the
error:
>
> with fs.open_input_stream(read_path, **open_stream_args) as f:
>   File "pyarrow/_fs.pyx", line 627, in
> pyarrow._fs.FileSystem.open_input_stream
>   File "pyarrow/_fs.pyx", line 557, in
> pyarrow._fs.FileSystem._wrap_input_stream
>   File "pyarrow/io.pxi", line 1283, in
> pyarrow.lib.CompressedInputStream.__init__
>   File "pyarrow/error.pxi", line 143, in
> pyarrow.lib.pyarrow_internal_check_status
>   File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
> pyarrow.lib.ArrowNotImplementedError: Streaming decompression unsupported
> with Snappy
>

Can anyone plz help me with this?

Thanks,
Jialin

[Computer-go] [cfp] General Video Game AI Learning Competition @IEEE CoG2021

[Jialin Liu]

Apologies for cross-posting

*[Call for participants] GVGAI Learning Competition@IEEE CoG2021*

Submission deadline: 31 July, 2021
Competition webpage: https://github.com/SUSTechGameAI/GVGAI_GYM
Competition framework: http://aingames.cn/gvgai/cog2021/
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[jira] [Commented] (HDFS-12571) Ozone: remove spaces from the beginning of the hdfs script

[Jialin Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-12571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125635#comment-17125635
 ] 

Jialin Liu commented on HDFS-12571:
---

as of June 14 2020, I'm still seeing the same issue:
{code:java}
sudo bash start-all.sh 
Password:
Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh: 
line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh: 
line 398: `  done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
249: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
257: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 218: 
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 219: 
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
268: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
275: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
278: hadoop_generic_java_subcmd_handler: command not found
Starting datanodes
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh: 
line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh: 
line 398: `  done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
249: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
257: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 218: 
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 219: 
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
268: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
275: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
278: hadoop_generic_java_subcmd_handler: command not found
Starting secondary namenodes [Jialin.local]
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh: 
line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh: 
line 398: `  done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
249: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
257: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 218: 
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 219: 
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line 
268: hadoop_ad

[Computer-go] CFP: WCCI/CEC2020 Special Session on Games - Extended submission deadline: 30 Jan 2020

[Jialin Liu]

*Apologies for cross-posting*

*CFP of WCCI/CEC2020 Special Session on Games*:
http://www.liujialin.tech/events/wcci2020-games.html

*Extended submission deadline: 30 Jan 2020*
Submission link: https://ieee-cis.org/conferences/cec2020/upload.php
Please select S02. Special Session on Games
 as main topic when
submitting papers.

*Call for participants of competitions at WCCI/CEC2020: *
CEC-C6 EvoMan: General Game-playing Agent Competition

http://pesquisa.ufabc.edu.br/hal/Evoman.html
Deadline: March 31st, 23:59 (GMT)

CEC-C8 Strategy Card Game AI Competition

https://jakubkowalski.tech/Projects/LOCM/CEC20/
Deadline: 24th May 2020, 23:59 (GMT)

More competitions can be found at https://wcci2020.org/competitions/
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Computer-go] CFP: WCCI/CEC2020 Special Session on Games - Submission deadline: 15 Jan 2020

[Jialin Liu]

*Apologies for cross-posting*

*CFP of WCCI/CEC2020 Special Session on Games*:
http://www.liujialin.tech/events/wcci2020-games.html
Submission deadline: 15 Jan 2020
Please select CEC-21 Special Session on Games
 when submitting
papers.

*Call for participants of competitions at WCCI/CEC2020: *
CEC-C6 EvoMan: General Game-playing Agent Competition

http://pesquisa.ufabc.edu.br/hal/Evoman.html
Deadline: March 31st, 23:59 (GMT)
CEC-C8 Strategy Card Game AI Competition

https://jakubkowalski.tech/Projects/LOCM/CEC20/
Deadline: 24th May 2020, 23:59 (GMT)

More competitions can be found at https://wcci2020.org/competitions/
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Computer-go] [CFP] Competition and Benchmark Track at 2019 IEEE Conference on Games (former IEEE CIG)

[Jialin Liu]

Dear all,

Apologies for cross posting. I just wanted to draw your attention to the
``Competition and Benchmark'' track at this year's IEEE Conference on Games
(http://ieee-cog.org/).

This track invites submissions where the main topic is a game or
game-related benchmark or competition. Submission can be new competition
proposals, novel approaches to existing competitions, or critical
commentary on competitions.

Topics include but are not limited to the following:

   - Evaluation of AI agents.
   - Machine learning related to games (e.g. for predicting player or AI
   behaviour).
   - Learning forward models.
   - Assessment of procedural content generators.
   - Analysis tools for existing competitions.
   - Game Competitions to more general AI.

If you are working on, organizing, or planning a competition and you want
to write a scientific paper about it, this could be a good venue for it.


*The deadline for short, vision, demo and COMPETITION papers is the 14th of
May, 2019. *

Have a look, and best regards, and possibly see you in London.

Christoph Salge & Jialin Liu
Competition and Benchmark Track Chairs
The 2019 IEEE Conference on Games
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Computer-go] [CFP] Special Session on Games@IEEE CEC2019 (Extended deadline: 31 Jan 2019)

[Jialin Liu]

[Apologies if you receive multiple copies of this CFP]

*Call for Papers - Special Session on Games*
*The 2019 IEEE Congress on Evolutionary Computation (IEEE CEC2019)*
*Wellington, New Zealand, 10-13 June 2019*
*http://cec2019.org/programs/special_sessions.html#cec-04
<http://cec2019.org/programs/special_sessions.html#cec-04>*

*Important Dates*
*Extended submission deadline: 31 January 2019*
Notification: 7 March 2019
Final paper submission: 31 March 2019

*Please select "CEC-04: Special Session on Games" when submitting your
paper.*
**This special session is organized in association with the **IEEE CIS
Computational Intelligence Society Technical Committee on Games (Game TC)*
*.*


Games are an ideal domain to study computational intelligence (CI) methods
because they provide affordable, competitive, dynamic, reproducible
environments suitable for testing new search algorithms, pattern-based
evaluation methods, or learning concepts. Games scale from simple problems
for developing algorithms to incredibly hard problems for testing
algorithms to the limit. They are also interesting to observe, fun to play,
and very attractive to students. Additionally, there is great potential for
CI methods to improve the design and development of both computer games as
well as tabletop games, board games, and puzzles. This special session aims
at gathering leaders and neophytes in games research as well as
practitioners in this field who research applications of computational
intelligence methods to computer games.

*Topics*
In general, papers are welcome that consider all kinds of applications of
methods (evolutionary computation, supervised learning, unsupervised
learning, fuzzy systems, game-tree search, rolling horizon algorithms,
MCTS, etc.) to games (card games, board games, mathematical games, action
games, strategy games, role-playing games, arcade games, serious games,
etc.).
Examples include but are not limited to:


   - Adaptation in games
   - Automatic game testing
   - Coevolution in games
   - Comparative studies (e.g. CI versus human-designed players)
   - Dynamic difficulty in games
   - Games as test-beds for algorithms
   - Imitating human players
   - Learning to play games
   - Multi-agent and multi-strategy learning
   - Player/Opponent modelling
   - Procedural content generation
   - CI for Serious Games (e.g., games for health care, education or
   training)
   - Results of game-based CI and open competitions



*Organizers*
*Jialin Liu, liujl(at)sustc.edu.cn <http://sustc.edu.cn/>*
Research Assistant Professor, Dept. of Computer Science and Engineering,
Southern University of Science and Technology, China
*Daniel Ashlock, dashlock(at)uoguelph.ca <http://uoguelph.ca/>*
Professor, Dept. of Mathematics and Statistics, University of Guelph, Canada
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Computer-go] [CFP] Special Session on Games@IEEE CEC2019 (Extended deadline: 21 Jan 2019)

[Jialin Liu]

[Apologies if you receive multiple copies of this CFP]

*Call for Papers - Special Session on Games*
*The 2019 IEEE Congress on Evolutionary Computation (IEEE CEC2019)*
*Wellington, New Zealand, 10-13 June 2019*
*http://cec2019.org/programs/special_sessions.html#cec-04
<http://cec2019.org/programs/special_sessions.html#cec-04>*

*Important Dates*
*Extended submission deadline: 21 January 2019*
Notification: 7 March 2019
Final paper submission: 31 March 2019

*Please select "CEC-04: Special Session on Games" when submitting your
paper.*
**This special session is organized in association with the **IEEE CIS
Computational Intelligence Society Technical Committee on Games (Game TC)*
*.*


Games are an ideal domain to study computational intelligence (CI) methods
because they provide affordable, competitive, dynamic, reproducible
environments suitable for testing new search algorithms, pattern-based
evaluation methods, or learning concepts. Games scale from simple problems
for developing algorithms to incredibly hard problems for testing
algorithms to the limit. They are also interesting to observe, fun to play,
and very attractive to students. Additionally, there is great potential for
CI methods to improve the design and development of both computer games as
well as tabletop games, board games, and puzzles. This special session aims
at gathering leaders and neophytes in games research as well as
practitioners in this field who research applications of computational
intelligence methods to computer games.

*Topics*
In general, papers are welcome that consider all kinds of applications of
methods (evolutionary computation, supervised learning, unsupervised
learning, fuzzy systems, game-tree search, rolling horizon algorithms,
MCTS, etc.) to games (card games, board games, mathematical games, action
games, strategy games, role-playing games, arcade games, serious games,
etc.).
Examples include but are not limited to:


   - Adaptation in games
   - Automatic game testing
   - Coevolution in games
   - Comparative studies (e.g. CI versus human-designed players)
   - Dynamic difficulty in games
   - Games as test-beds for algorithms
   - Imitating human players
   - Learning to play games
   - Multi-agent and multi-strategy learning
   - Player/Opponent modelling
   - Procedural content generation
   - CI for Serious Games (e.g., games for health care, education or
   training)
   - Results of game-based CI and open competitions



*Organizers*
*Jialin Liu, liujl(at)sustc.edu.cn <http://sustc.edu.cn/>*
Research Assistant Professor, Dept. of Computer Science and Engineering,
Southern University of Science and Technology, China
*Daniel Ashlock, dashlock(at)uoguelph.ca <http://uoguelph.ca/>*
Professor, Dept. of Mathematics and Statistics, University of Guelph, Canada
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[jira] [Commented] (SPARK-26261) Spark does not check completeness temporary file

[Jialin LIu (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-26261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710883#comment-16710883
 ] 

Jialin LIu commented on SPARK-26261:


Our initial test is:

We start a word count workflow including persisting blocks to disk. After we 
make sure that there are some blocks on the disk already, we use the truncate 
command to truncate part of the block. We compare the result with the result 
produced by workflow without fault injection. 

> Spark does not check completeness temporary file 
> -
>
> Key: SPARK-26261
> URL: https://issues.apache.org/jira/browse/SPARK-26261
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.2
>    Reporter: Jialin LIu
>Priority: Minor
>
> Spark does not check temporary files' completeness. When persisting to disk 
> is enabled on some RDDs, a bunch of temporary files will be created on 
> blockmgr folder. Block manager is able to detect missing blocks while it is 
> not able detect file content being modified during execution. 
> Our initial test shows that if we truncate the block file before being used 
> by executors, the program will finish without detecting any error, but the 
> result content is totally wrong.
> We believe there should be a file checksum on every RDD file block and these 
> files should be protected by checksum.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-26261) Spark does not check completeness temporary file

[Jialin LIu (JIRA)]

Jialin LIu created SPARK-26261:
--

 Summary: Spark does not check completeness temporary file 
 Key: SPARK-26261
 URL: https://issues.apache.org/jira/browse/SPARK-26261
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 2.3.2
Reporter: Jialin LIu


Spark does not check temporary files' completeness. When persisting to disk is 
enabled on some RDDs, a bunch of temporary files will be created on blockmgr 
folder. Block manager is able to detect missing blocks while it is not able 
detect file content being modified during execution. 

Our initial test shows that if we truncate the block file before being used by 
executors, the program will finish without detecting any error, but the result 
content is totally wrong.

We believe there should be a file checksum on every RDD file block and these 
files should be protected by checksum.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-26197) Spark master fails to detect driver process pause

[Jialin LIu (JIRA)]

Jialin LIu created SPARK-26197:
--

 Summary: Spark master fails to detect driver process pause
 Key: SPARK-26197
 URL: https://issues.apache.org/jira/browse/SPARK-26197
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 2.3.2
Reporter: Jialin LIu


I was using Spark 2.3.2 with standalone cluster and submit job using cluster 
mode. After I submit the job, I deliberately pause the driver process 
(throughout shell command "kill -stop (driver process id) ") to see if the 
master can detect this problem. The result shows that the driver will never 
stop. All the executors will try to talk back to driver and will give up in 10 
minutes. Master can detect executor failures and try to reassign new executor 
process to redo the job. New executor will try to create RPC connection with 
driver and will fail in 2 minutes. Master will endlessly spawn new executors 
without detecting driver failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[Computer-go] [CFP] Special Session on Games@IEEE CEC2019, Wellington, New Zealand (10-13 June 2019)

[Jialin Liu]

[Apologies if you receive multiple copies of this CFP]


*Call for Papers - Special Session on Games*

*The 2019 IEEE Congress on Evolutionary Computation (IEEE CEC2019)*

*Wellington, New Zealand, 10-13 June 2019*

*http://cec2019.org/programs/special_sessions.html#cec-04
<http://cec2019.org/programs/special_sessions.html#cec-04>*


*Important Dates*

Submission deadline: 7 January 2019

Notification: 7 March 2019

Final paper submission: 31 March 2019


*Please select "CEC-04: Special Session on Games" when submitting your
paper.*

**This special session is organized in association with the **IEEE CIS
Computational Intelligence Society Technical Committee on Games (Game TC)*
*.*



Games are an ideal domain to study computational intelligence (CI) methods
because they provide affordable, competitive, dynamic, reproducible
environments suitable for testing new search algorithms, pattern-based
evaluation methods, or learning concepts. Games scale from simple problems
for developing algorithms to incredibly hard problems for testing
algorithms to the limit. They are also interesting to observe, fun to play,
and very attractive to students. Additionally, there is great potential for
CI methods to improve the design and development of both computer games as
well as tabletop games, board games, and puzzles. This special session aims
at gathering leaders and neophytes in games research as well as
practitioners in this field who research applications of computational
intelligence methods to computer games.


*Topics*

In general, papers are welcome that consider all kinds of applications of
methods (evolutionary computation, supervised learning, unsupervised
learning, fuzzy systems, game-tree search, rolling horizon algorithms,
MCTS, etc.) to games (card games, board games, mathematical games, action
games, strategy games, role-playing games, arcade games, serious games,
etc.).

Examples include but are not limited to:


   - Adaptation in games
   - Automatic game testing
   - Coevolution in games
   - Comparative studies (e.g. CI versus human-designed players)
   - Dynamic difficulty in games
   - Games as test-beds for algorithms
   - Imitating human players
   - Learning to play games
   - Multi-agent and multi-strategy learning
   - Player/Opponent modelling
   - Procedural content generation
   - CI for Serious Games (e.g., games for health care, education or
   training)
   - Results of game-based CI and open competitions


*Organizers*

*Jialin Liu, liujl(at)sustc.edu.cn <http://sustc.edu.cn>*

Research Assistant Professor, Dept. of Computer Science and Engineering,
Southern University of Science and Technology, China

*Daniel Ashlock, dashlock(at)uoguelph.ca <http://uoguelph.ca>*

Professor, Dept. of Mathematics and Statistics, University of Guelph, Canada
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Openstack] error in deleting objects

[Jialin Liu]

Hi,
When I tried to delete all objects, with command 'delete -a', it seems some
container are not found:

Error Deleting: some.h5_9958: u"Container u'some.h5_9958' not found"
container:some.h5_9958

Does anyone know how to deal with this?

Best,
Jialin
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [ceph-users] IO to OSD with librados

[Jialin Liu]

Thanks for the advice, Dan.

I'll try to reconfigure the cluster and see if the performance changes.

Best,
Jialin

On Tue, Jun 19, 2018 at 12:02 AM Dan van der Ster 
wrote:

> On Tue, Jun 19, 2018 at 1:04 AM Jialin Liu  wrote:
> >
> > Hi Dan, Thanks for the follow-ups.
> >
> > I have just tried running multiple librados MPI applications from
> multiple nodes, it does show increased bandwidth,
> > with ceph -w, I observed as high as 500MB/sec (previously only 160MB/sec
> ), I think I can do finer tuning by
> > coordinating more concurrent applications to get the peak. (Sorry, I
> only have one node having rados cli installed, so I can't follow your
> example to stress the server)
> >
> >> Then you can try different replication or erasure coding settings to
> >> learn their impact on performance...
> >
> >
> > Good points.
> >
> >>
> >> PPS. What are those 21.8TB devices ?
> >
> >
> > The storage arrays are Nexsan E60 arrays having two active-active
> redundant
> > controllers, 60 3 TB disk drives. The disk drives are organized into six
> 8+2
> > Raid 6 LUNs of 24 TB each.
> >
>
> This is not the ideal Ceph hardware. Ceph is designed to use disks
> directly -- JBODs. All redundancy is handled at the RADOS level, so
> you can happily save lots of cash on your servers. I suggest reading
> through the various Ceph hardware recommendations that you can find
> via Google.
>
> I can't tell from here if this is the root cause of your performance
> issue -- but you should plan future clusters to use JBODs instead of
> expensive arrays.
>
> >
> >>
> >> PPPS. Any reason you are running jewel instead of luminous or mimic?
> >
> >
> > I have to ask the cluster admin, I'm not sure about it.
> >
> > I have one more questions, regarding the OSD server and OSDs, I was told
> that the IO has to go through the 4 OSD servers (hosts), before touching
> the OSDs,
> > This is confusing to me, as I learned from the ceph document
> http://docs.ceph.com/docs/jewel/rados/operations/monitoring-osd-pg/#monitoring-osds
> > the librados can talk to the OSDs directly, what am I missing here?
>
> You should have one ceph-osd process per disk (or per LUN in your
> case). The clients connect to the ceph-osd processes directly.
>
> -- dan
>
>
> >
> >
> > Best,
> > Jialin
> > NERSC/LBNL
> >
> >
> >>
> >> On Mon, Jun 18, 2018 at 3:43 PM Jialin Liu  wrote:
> >> >
> >> > Hi, To make the the problem clearer, here is the configuration of the
> cluster:
> >> >
> >> > The 'problem' I have is the low bandwidth no matter how I increase
> the concurrency.
> >> > I have tried using MPI to launch 322 processes, each calling librados
> to create a handle and initialize the io context, and write one 80MB object.
> >> > I only got ~160 MB/sec, with one process, I can get ~40 MB/sec, I'm
> wondering if the number of client-osd connection is limited by the number
> of hosts.
> >> >
> >> > Best,
> >> > Jialin
> >> > NERSC/LBNL
> >> >
> >> > $ceph osd tree
> >> >
> >> > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >> >
> >> > -1 1047.59473 root default
> >> >
> >> > -2  261.89868 host ngfdv036
> >> >
> >> >  0   21.82489 osd.0  up  1.0  1.0
> >> >
> >> >  4   21.82489 osd.4  up  1.0  1.0
> >> >
> >> >  8   21.82489 osd.8  up  1.0  1.0
> >> >
> >> > 12   21.82489 osd.12 up  1.0  1.0
> >> >
> >> > 16   21.82489 osd.16 up  1.0  1.0
> >> >
> >> > 20   21.82489 osd.20 up  1.0  1.0
> >> >
> >> > 24   21.82489 osd.24 up  1.0  1.0
> >> >
> >> > 28   21.82489 osd.28 up  1.0  1.0
> >> >
> >> > 32   21.82489 osd.32 up  1.0  1.0
> >> >
> >> > 36   21.82489 osd.36 up  1.0  1.0
> >> >
> >> > 40   21.82489 osd.40 up  1.0  1.0
> >> >
> >> > 44   21.82489 osd.44 up  1.0  1.0
> >> >
> >> > -3 

Re: [ceph-users] IO to OSD with librados

[Jialin Liu]

Hi Dan, Thanks for the follow-ups.

I have just tried running multiple librados MPI applications from multiple
nodes, it does show increased bandwidth,
with ceph -w, I observed as high as 500MB/sec (previously only 160MB/sec ),
I think I can do finer tuning by
coordinating more concurrent applications to get the peak. (Sorry, I only
have one node having rados cli installed, so I can't follow your example to
stress the server)

Then you can try different replication or erasure coding settings to
> learn their impact on performance...
>

Good points.


> PPS. What are those 21.8TB devices ?
>

The storage arrays are Nexsan E60 arrays having two active-active redundant
controllers, 60 3 TB disk drives. The disk drives are organized into six 8+2
Raid 6 LUNs of 24 TB each.



> PPPS. Any reason you are running jewel instead of luminous or mimic?
>

I have to ask the cluster admin, I'm not sure about it.

I have one more questions, regarding the OSD server and OSDs, I was told
that the IO has to go through the 4 OSD servers (hosts), before touching
the OSDs,
This is confusing to me, as I learned from the ceph document
http://docs.ceph.com/docs/jewel/rados/operations/monitoring-osd-pg/#monitoring-osds
the librados can talk to the OSDs directly, what am I missing here?


Best,
Jialin
NERSC/LBNL



> On Mon, Jun 18, 2018 at 3:43 PM Jialin Liu  wrote:
> >
> > Hi, To make the the problem clearer, here is the configuration of the
> cluster:
> >
> > The 'problem' I have is the low bandwidth no matter how I increase the
> concurrency.
> > I have tried using MPI to launch 322 processes, each calling librados to
> create a handle and initialize the io context, and write one 80MB object.
> > I only got ~160 MB/sec, with one process, I can get ~40 MB/sec, I'm
> wondering if the number of client-osd connection is limited by the number
> of hosts.
> >
> > Best,
> > Jialin
> > NERSC/LBNL
> >
> > $ceph osd tree
> >
> > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >
> > -1 1047.59473 root default
> >
> > -2  261.89868 host ngfdv036
> >
> >  0   21.82489 osd.0  up  1.0  1.0
> >
> >  4   21.82489 osd.4  up  1.0  1.0
> >
> >  8   21.82489 osd.8  up  1.0  1.0
> >
> > 12   21.82489 osd.12 up  1.0  1.0
> >
> > 16   21.82489 osd.16 up  1.0  1.0
> >
> > 20   21.82489 osd.20 up  1.0  1.0
> >
> > 24   21.82489 osd.24 up  1.0  1.0
> >
> > 28   21.82489 osd.28 up  1.0  1.0
> >
> > 32   21.82489 osd.32 up  1.0  1.0
> >
> > 36   21.82489 osd.36 up  1.0  1.0
> >
> > 40   21.82489 osd.40 up  1.0  1.0
> >
> > 44   21.82489 osd.44 up  1.0  1.0
> >
> > -3  261.89868 host ngfdv037
> >
> >  1   21.82489 osd.1  up  1.0  1.0
> >
> >  5   21.82489 osd.5  up  1.0  1.0
> >
> >  9   21.82489 osd.9  up  1.0  1.0
> >
> > 13   21.82489 osd.13 up  1.0  1.0
> >
> > 17   21.82489 osd.17 up  1.0  1.0
> >
> > 21   21.82489 osd.21 up  1.0  1.0
> >
> > 25   21.82489 osd.25 up  1.0  1.0
> >
> > 29   21.82489 osd.29 up  1.0  1.0
> >
> > 33   21.82489 osd.33 up  1.0  1.0
> >
> > 37   21.82489 osd.37 up  1.0  1.0
> >
> > 41   21.82489 osd.41 up  1.0  1.0
> >
> > 45   21.82489 osd.45 up  1.0  1.0
> >
> > -4  261.89868 host ngfdv038
> >
> >  2   21.82489 osd.2  up  1.0  1.0
> >
> >  6   21.82489 osd.6  up  1.0  1.0
> >
> > 10   21.82489 osd.10 up  1.0  1.0
> >
> > 14   21.82489 osd.14 up  1.0  1.0
> >
> > 18   21.82489 osd.18 up  1.0  1.0
> >
> > 22   21.82489 osd.22 up  1.0  1.0
> >
> > 26   21.82489 osd.26 up  1.0  1.0

Re: [ceph-users] IO to OSD with librados

[Jialin Liu]

Hi, To make the the problem clearer, here is the configuration of the
cluster:

The 'problem' I have is the low bandwidth no matter how I increase the
concurrency.
I have tried using MPI to launch 322 processes, each calling librados to
create a handle and initialize the io context, and write one 80MB object.
I only got ~160 MB/sec, with one process, I can get ~40 MB/sec, I'm
wondering if the number of client-osd connection is limited by the number
of hosts.

Best,
Jialin
NERSC/LBNL

$ceph osd tree

ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 1047.59473 root default

-2  261.89868 host ngfdv036

 0   21.82489 osd.0  up  1.0  1.0

 4   21.82489 osd.4  up  1.0  1.0

 8   21.82489 osd.8  up  1.0  1.0

12   21.82489 osd.12 up  1.0  1.0

16   21.82489 osd.16 up  1.0  1.0

20   21.82489 osd.20 up  1.0  1.0

24   21.82489 osd.24 up  1.0  1.0

28   21.82489 osd.28 up  1.0  1.0

32   21.82489 osd.32 up  1.0  1.0

36   21.82489 osd.36 up  1.0  1.0

40   21.82489 osd.40 up  1.0  1.0

44   21.82489 osd.44 up  1.0  1.0

-3  261.89868 host ngfdv037

 1   21.82489 osd.1  up  1.0  1.0

 5   21.82489 osd.5  up  1.0  1.0

 9   21.82489 osd.9  up  1.0  1.0

13   21.82489 osd.13 up  1.0  1.0

17   21.82489 osd.17 up  1.0  1.0

21   21.82489 osd.21 up  1.0  1.0

25   21.82489 osd.25 up  1.0  1.0

29   21.82489 osd.29 up  1.0  1.0

33   21.82489 osd.33 up  1.0  1.0

37   21.82489 osd.37 up  1.0  1.0

41   21.82489 osd.41 up  1.0  1.0

45   21.82489 osd.45 up  1.0  1.0

-4  261.89868 host ngfdv038

 2   21.82489 osd.2  up  1.0  1.0

 6   21.82489 osd.6  up  1.0  1.0

10   21.82489 osd.10 up  1.0  1.0

14   21.82489 osd.14 up  1.0  1.0

18   21.82489 osd.18 up  1.0  1.0

22   21.82489 osd.22 up  1.0  1.0

26   21.82489 osd.26 up  1.0  1.0

30   21.82489 osd.30 up  1.0  1.0

34   21.82489 osd.34 up  1.0  1.0

38   21.82489 osd.38 up  1.0  1.0

42   21.82489 osd.42 up  1.0  1.0

46   21.82489 osd.46 up  1.0  1.0

-5  261.89868 host ngfdv039

 3   21.82489 osd.3  up  1.0  1.0

 7   21.82489 osd.7  up  1.0  1.0

11   21.82489 osd.11 up  1.0  1.0

15   21.82489 osd.15 up  1.0  1.0

19   21.82489 osd.19 up  1.0  1.0

23   21.82489 osd.23 up  1.0  1.0

27   21.82489 osd.27 up  1.0  1.0

31   21.82489 osd.31 up  1.0  1.0

35   21.82489 osd.35 up  1.0  1.0

39   21.82489 osd.39 up  1.0  1.0

43   21.82489 osd.43 up  1.0  1.0

47   21.82489 osd.47 up  1.0  1.0

ceph -s

cluster 2b0e2d2b-3f63-4815-908a-b032c7f9427a

 health HEALTH_OK

 monmap e1: 2 mons at
{ngfdv076=128.55.xxx.xx:6789/0,ngfdv078=128.55.xxx.xx:6789/0}

election epoch 4, quorum 0,1 ngfdv076,ngfdv078

 osdmap e280: 48 osds: 48 up, 48 in

flags sortbitwise,require_jewel_osds

  pgmap v117283: 3136 pgs, 11 pools, 25600 MB data, 510 objects

79218 MB used, 1047 TB / 1047 TB avail

3136 active+clean


On Mon, Jun 18, 2018 at 1:06 AM Jialin Liu  wrote:

> Thank you Dan. I’ll try it.
>
> Best,
> Jialin
> NERSC/LBNL
>
> > On Jun 18, 2018, at 12:22 AM, Dan van der Ster 
> wrote:
> >
> > Hi,
> >
> > One way you can see exactly what is happening when you write an object
> > is with --debug_ms=1.
> >
> > For example, I write a 100MB object to a test pool:  rados
> > --debug_ms=1 -p test put 100M.dat 100M.dat
> > I pasted the output of this here: https://pastebin.com/Zg8rjaTV
> > In 

Re: [ceph-users] IO to OSD with librados

[Jialin Liu]

Thank you Dan. I’ll try it.

Best,
Jialin
NERSC/LBNL

> On Jun 18, 2018, at 12:22 AM, Dan van der Ster  wrote:
> 
> Hi,
> 
> One way you can see exactly what is happening when you write an object
> is with --debug_ms=1.
> 
> For example, I write a 100MB object to a test pool:  rados
> --debug_ms=1 -p test put 100M.dat 100M.dat
> I pasted the output of this here: https://pastebin.com/Zg8rjaTV
> In this case, it first gets the cluster maps from a mon, then writes
> the object to osd.58, which is the primary osd for PG 119.77:
> 
> # ceph pg 119.77 query | jq .up
> [
>  58,
>  49,
>  31
> ]
> 
> Otherwise I answered your questions below...
> 
>> On Sun, Jun 17, 2018 at 8:29 PM Jialin Liu  wrote:
>> 
>> Hello,
>> 
>> I have a couple questions regarding the IO on OSD via librados.
>> 
>> 
>> 1. How to check which osd is receiving data?
>> 
> 
> See `ceph osd map`.
> For my example above:
> 
> # ceph osd map test 100M.dat
> osdmap e236396 pool 'test' (119) object '100M.dat' -> pg 119.864b0b77
> (119.77) -> up ([58,49,31], p58) acting ([58,49,31], p58)
> 
>> 2. Can the write operation return immediately to the application once the 
>> write to the primary OSD is done? or does it return only when the data is 
>> replicated twice? (size=3)
> 
> Write returns once it is safe on *all* replicas or EC chunks.
> 
>> 3. What is the I/O size in the lower level in librados, e.g., if I send a 
>> 100MB request with 1 thread, does librados send the data by a fixed 
>> transaction size?
> 
> This depends on the client. The `rados` CLI example I showed you broke
> the 100MB object into 4MB parts.
> Most use-cases keep the objects around 4MB or 8MB.
> 
>> 4. I have 4 OSS, 48 OSDs, will the 4 OSS become the bottleneck? from the 
>> ceph documentation, once the cluster map is received by the client, the 
>> client can talk to OSD directly, so the assumption is the max parallelism 
>> depends on the number of OSDs, is this correct?
>> 
> 
> That's more or less correct -- the IOPS and BW capacity of the cluster
> generally scales linearly with number of OSDs.
> 
> Cheers,
> Dan
> CERN
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] IO to OSD with librados

[Jialin Liu]

Sorry about the misused term 'OSS: object storage server' (a term often
used in Lustre filesystem), what I meant is 4 hosts, each manages 12 OSDs.
Thanks for anyone who may answer any of my questions.

Best,
Jialin
NERSC/LBNL

On Sun, Jun 17, 2018 at 11:29 AM Jialin Liu  wrote:

> Hello,
>
> I have a couple questions regarding the IO on OSD via librados.
>
>
> 1. How to check which osd is receiving data?
>
> 2. Can the write operation return immediately to the application once the
> write to the primary OSD is done? or does it return only when the data is
> replicated twice? (size=3)
>
> 3. What is the I/O size in the lower level in librados, e.g., if I send a
> 100MB request with 1 thread, does librados send the data by a fixed
> transaction size?
>
> 4. I have 4 OSS, 48 OSDs, will the 4 OSS become the bottleneck? from the
> ceph documentation, once the cluster map is received by the client, the
> client can talk to OSD directly, so the assumption is the max parallelism
> depends on the number of OSDs, is this correct?
>
>
> Best,
>
> Jialin
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] IO to OSD with librados

[Jialin Liu]

Hello,

I have a couple questions regarding the IO on OSD via librados.


1. How to check which osd is receiving data?

2. Can the write operation return immediately to the application once the
write to the primary OSD is done? or does it return only when the data is
replicated twice? (size=3)

3. What is the I/O size in the lower level in librados, e.g., if I send a
100MB request with 1 thread, does librados send the data by a fixed
transaction size?

4. I have 4 OSS, 48 OSDs, will the 4 OSS become the bottleneck? from the
ceph documentation, once the cluster map is received by the client, the
client can talk to OSD directly, so the assumption is the max parallelism
depends on the number of OSDs, is this correct?


Best,

Jialin
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to build libradosstriper

[Jialin Liu]

Hi Guys,

The striping seems to be slightly better than non-striping write, given
that my storage is configured with 4OSS, and 48OSD. each OSD is 8+2 Raid 6
of 24TB capacity.
But still the performance is around 100MB/sec. On a single haswell core,
I'm able to get 1GB/sec with dd buffered IO.

My questions are:
1. Does librados use direct IO?
2. Is there option to leverage page cache?
3. What is the strategy to get the peak bandwidth on a rados object store?
4. Can I use MPI for parallel IO?

Best,
Jialin
NERSC

On Wed, May 30, 2018 at 11:08 AM, Jialin Liu  wrote:

> Thanks Kefu.
>
> Best,
> Jialin
> NERSC
>
> On Tue, May 29, 2018 at 11:52 PM, kefu chai  wrote:
>
>> On Wed, May 30, 2018 at 11:53 AM, Jialin Liu  wrote:
>> > Hi Brad,
>> >
>> > You are correct. the librados.so has the symbol but what I copied is a
>> wrong
>> > file.
>> > Now I can test the striper api with the previous C example and this cpp
>> > example:
>> > http://mrbojangles3.github.io/ceph/systems/striping/alignmen
>> t/2017/05/28/Ceph-Stripe/
>> > Both are working, but I haven't got any performance number yet.
>> > ps. I also found that the stripe unit must be set before object_size
>> > properly.
>> >
>> > Thanks much.
>> >
>> > Best,
>> > Jialin
>> > NERSC
>> >
>> >
>> > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard 
>> wrote:
>> >>
>> >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu  wrote:
>> >> > Thanks Brad,
>> >> > I run nm on those .so file, it prints 'no symbol'
>> >>
>> >> OK, well you need to link to a library that exports that symbol (has
>> >> it defined in its Text section). I suspect you'll find it is defined
>> >> in libceph-common.so so try linking to that explicitly.
>>
>> please note, libceph-common does not exist in jewel (i.e. v10.2.10)
>> yet. it was introduce in luminous. and libceph-common is not supposed
>> to be linked against by rados application directly. it is an internal
>> library and is linked by librados, libradosstriper and other rados
>> tools/daemons.
>>
>> >>
>> >> >
>> >> > Then with ldd librados.so, I don't see the libceph-common.so,
>> instead:
>> >> >>
>> >> >> jialin@cori12: ldd librados.so
>> >> >> linux-vdso.so.1 (0x2aacf000)
>> >> >> libboost_thread-mt.so.1.53.0
>> >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0
>> (0x2d24d000)
>> >> >> libboost_system-mt.so.1.53.0 =>
>> >> >> /rados_install/lib/libboost_system-mt.so.1.53.0
>> (0x2d464000)
>> >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
>> >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
>> >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
>> >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
>> >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
>> >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
>> >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
>> >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
>> >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
>> >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
>> >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
>> >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
>> >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
>> >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
>> >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
>> >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
>> >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)
>> >>
>> >> Things were obviously different in jewel but the concept remains the
>> >> same. For each external symbol your binary requires you need to link
>> >> to a library that exports that symbol so finding the library that
>> >> exports the symbols in the error message should resolve the error.
>> >>
>> >> >
>> >> >
>> >> > I also found this thread: http://tracker.ceph.com/issues/14788
>> >>

Re: [ceph-users] how to build libradosstriper

[Jialin Liu]

Thanks Kefu.

Best,
Jialin
NERSC

On Tue, May 29, 2018 at 11:52 PM, kefu chai  wrote:

> On Wed, May 30, 2018 at 11:53 AM, Jialin Liu  wrote:
> > Hi Brad,
> >
> > You are correct. the librados.so has the symbol but what I copied is a
> wrong
> > file.
> > Now I can test the striper api with the previous C example and this cpp
> > example:
> > http://mrbojangles3.github.io/ceph/systems/striping/
> alignment/2017/05/28/Ceph-Stripe/
> > Both are working, but I haven't got any performance number yet.
> > ps. I also found that the stripe unit must be set before object_size
> > properly.
> >
> > Thanks much.
> >
> > Best,
> > Jialin
> > NERSC
> >
> >
> > On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard 
> wrote:
> >>
> >> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu  wrote:
> >> > Thanks Brad,
> >> > I run nm on those .so file, it prints 'no symbol'
> >>
> >> OK, well you need to link to a library that exports that symbol (has
> >> it defined in its Text section). I suspect you'll find it is defined
> >> in libceph-common.so so try linking to that explicitly.
>
> please note, libceph-common does not exist in jewel (i.e. v10.2.10)
> yet. it was introduce in luminous. and libceph-common is not supposed
> to be linked against by rados application directly. it is an internal
> library and is linked by librados, libradosstriper and other rados
> tools/daemons.
>
> >>
> >> >
> >> > Then with ldd librados.so, I don't see the libceph-common.so, instead:
> >> >>
> >> >> jialin@cori12: ldd librados.so
> >> >> linux-vdso.so.1 (0x2aacf000)
> >> >> libboost_thread-mt.so.1.53.0
> >> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0
> (0x2d24d000)
> >> >> libboost_system-mt.so.1.53.0 =>
> >> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000)
> >> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
> >> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
> >> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
> >> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
> >> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
> >> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
> >> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
> >> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
> >> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
> >> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
> >> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
> >> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
> >> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
> >> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
> >> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
> >> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
> >> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)
> >>
> >> Things were obviously different in jewel but the concept remains the
> >> same. For each external symbol your binary requires you need to link
> >> to a library that exports that symbol so finding the library that
> >> exports the symbols in the error message should resolve the error.
> >>
> >> >
> >> >
> >> > I also found this thread: http://tracker.ceph.com/issues/14788
> >> > which looks similar to the error I run into, and that thread mentioned
> >> > the
> >> > version between the headers file and the .so file, I think in my case
> >> > both
> >> > of them are v10.2.10, which should ok.
> >> >
> >> > Best,
> >> > Jialin,
> >> >
> >> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard 
> >> > wrote:
> >> >>
> >> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu 
> wrote:
> >> >> > Hi,
> >> >> > I'm trying to use the libradosstriper api, but having some trouble
> in
> >> >> > linking to lradosstriper. I copied only the `required' libraries
> from
> >> >> > an
> >> >> > pre-installed ceph (10.2.10), and put them under my local d

Re: [ceph-users] how to build libradosstriper

[Jialin Liu]

Hi Brad,

You are correct. the librados.so has the symbol but what I copied is a
wrong file.
Now I can test the striper api with the previous C example and this cpp
example:
http://mrbojangles3.github.io/ceph/systems/striping/alignment/2017/05/28/Ceph-Stripe/
Both are working, but I haven't got any performance number yet.
ps. I also found that the stripe unit must be set before object_size
properly.

Thanks much.

Best,
Jialin
NERSC


On Tue, May 29, 2018 at 7:11 PM, Brad Hubbard  wrote:

> On Wed, May 30, 2018 at 11:52 AM, Jialin Liu  wrote:
> > Thanks Brad,
> > I run nm on those .so file, it prints 'no symbol'
>
> OK, well you need to link to a library that exports that symbol (has
> it defined in its Text section). I suspect you'll find it is defined
> in libceph-common.so so try linking to that explicitly.
>
> >
> > Then with ldd librados.so, I don't see the libceph-common.so, instead:
> >>
> >> jialin@cori12: ldd librados.so
> >> linux-vdso.so.1 (0x2aacf000)
> >> libboost_thread-mt.so.1.53.0
> >> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000)
> >> libboost_system-mt.so.1.53.0 =>
> >> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000)
> >> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
> >> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
> >> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
> >> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
> >> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
> >> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
> >> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
> >> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
> >> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
> >> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
> >> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
> >> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
> >> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
> >> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
> >> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
> >> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)
>
> Things were obviously different in jewel but the concept remains the
> same. For each external symbol your binary requires you need to link
> to a library that exports that symbol so finding the library that
> exports the symbols in the error message should resolve the error.
>
> >
> >
> > I also found this thread: http://tracker.ceph.com/issues/14788
> > which looks similar to the error I run into, and that thread mentioned
> the
> > version between the headers file and the .so file, I think in my case
> both
> > of them are v10.2.10, which should ok.
> >
> > Best,
> > Jialin,
> >
> > On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard 
> wrote:
> >>
> >> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu  wrote:
> >> > Hi,
> >> > I'm trying to use the libradosstriper api, but having some trouble in
> >> > linking to lradosstriper. I copied only the `required' libraries from
> an
> >> > pre-installed ceph (10.2.10), and put them under my local directory
> >> > /rados_install/lib and rados_install/include, on a linux machine.
> >> >
> >> > /rados_install/lib:
> >> >>
> >> >> libboost_date_time.so.1.53.0  libboost_system.so.1.53.0
> >> >> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
> >> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0
> >> >> libboost_iostreams-mt.so.1.53.0
> >> >> libboost_iostreams.so.1.53.0
> >> >> libboost_random-mt.so.1.53.0  librados.so.2.0.0
> >> >> libboost_regex.so.1.53.0  librados_tp.so.2.0.0
> >> >> libboost_system-mt.so.1.53.0  libradosstriper.so.1.0.0
> >> >> libcephfs.so   librados.so  libradosstriper.so
> >> >
> >> >
> >> > /rados_install/include:
> >> >>
> >> >> |-- rados
> >> >> |   |-- buffer.h -> ../buffer.h
> >> >> |   |-- buffer_fwd.h -> ../buffer_fwd.h
> >> >> |   |-- crc32c.h -> ../crc32c.h
> >> >> |   |-- inline_memory.h -> ../inline_memory.h
> >> >> |   |-- librados.h
> >> >> |   |-- li

Re: [ceph-users] how to build libradosstriper

[Jialin Liu]

Thanks Brad,
I run nm on those .so file, it prints 'no symbol'

Then with ldd librados.so, I don't see the libceph-common.so, instead:

> jialin@cori12: ldd librados.so
> linux-vdso.so.1 (0x2aacf000)
> libboost_thread-mt.so.1.53.0
> =>/rados_install/lib/libboost_thread-mt.so.1.53.0 (0x2d24d000)
> libboost_system-mt.so.1.53.0 =>
> /rados_install/lib/libboost_system-mt.so.1.53.0 (0x2d464000)
> libssl3.so => /usr/lib64/libssl3.so (0x2d668000)
> libsmime3.so => /usr/lib64/libsmime3.so (0x2d8b4000)
> libnss3.so => /usr/lib64/libnss3.so (0x2dadb000)
> libnssutil3.so => /usr/lib64/libnssutil3.so (0x2ddfe000)
> libplds4.so => /usr/lib64/libplds4.so (0x2e02c000)
> libplc4.so => /usr/lib64/libplc4.so (0x2e23)
> libnspr4.so => /usr/lib64/libnspr4.so (0x2e435000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x2e673000)
> libdl.so.2 => /lib64/libdl.so.2 (0x2e89)
> libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x2ea94000)
> librt.so.1 => /lib64/librt.so.1 (0x2ec99000)
> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2eea1000)
> libm.so.6 => /lib64/libm.so.6 (0x2f22a000)
> libc.so.6 => /lib64/libc.so.6 (0x2f527000)
> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2f8c8000)
> libz.so.1 => /lib64/libz.so.1 (0x2fadf000)
> /lib64/ld-linux-x86-64.so.2 (0x2aaab000)


I also found this thread: http://tracker.ceph.com/issues/14788
which looks similar to the error I run into, and that thread mentioned the
version between the headers file and the .so file, I think in my case both
of them are v10.2.10, which should ok.

Best,
Jialin,

On Tue, May 29, 2018 at 6:17 PM, Brad Hubbard  wrote:

> On Wed, May 30, 2018 at 10:42 AM, Jialin Liu  wrote:
> > Hi,
> > I'm trying to use the libradosstriper api, but having some trouble in
> > linking to lradosstriper. I copied only the `required' libraries from an
> > pre-installed ceph (10.2.10), and put them under my local directory
> > /rados_install/lib and rados_install/include, on a linux machine.
> >
> > /rados_install/lib:
> >>
> >> libboost_date_time.so.1.53.0  libboost_system.so.1.53.0
> >> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
> >> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0
> >> libboost_iostreams-mt.so.1.53.0
> >> libboost_iostreams.so.1.53.0
> >> libboost_random-mt.so.1.53.0  librados.so.2.0.0
> >> libboost_regex.so.1.53.0  librados_tp.so.2.0.0
> >> libboost_system-mt.so.1.53.0  libradosstriper.so.1.0.0
> >> libcephfs.so   librados.so  libradosstriper.so
> >
> >
> > /rados_install/include:
> >>
> >> |-- rados
> >> |   |-- buffer.h -> ../buffer.h
> >> |   |-- buffer_fwd.h -> ../buffer_fwd.h
> >> |   |-- crc32c.h -> ../crc32c.h
> >> |   |-- inline_memory.h -> ../inline_memory.h
> >> |   |-- librados.h
> >> |   |-- librados.hpp
> >> |   |-- librgw.h
> >> |   |-- memory.h -> ../memory.h
> >> |   |-- objclass.h
> >> |   |-- page.h -> ../page.h
> >> |   |-- rados_types.h
> >> |   |-- rados_types.hpp
> >> |   `-- rgw_file.h
> >> `-- radosstriper
> >> |-- libradosstriper.h
> >> `-- libradosstriper.hpp
> >
> >
> > The test code is simple, as attached, and I compiled it with dynamic
> > linking:
> >>
> >> cc -L/rados_install/lib -I/rados_install/include -o librados_test
> >> librados_test.c -lrados -lradosstriper
> >
> >
> > But got the following message, can anyone plz advise?
> >
> >>
> >> /rados_install/lib/libradosstriper.so: undefined reference to
> >> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)'
> >> /rados_install/lib/libradosstriper.so: undefined reference to
> >> `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)'
> >> /rados_install/lib/libradosstriper.so: undefined reference to
> >> `librados::IoCtx::remove(std::string const&, int)'
> >> /rados_install/lib/libradosstriper.so: undefined reference to
> >> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned
> int)'
> >> /rados_install/lib/libradosstriper.so: undefined reference to
> >> `ceph::buffer::list::is_provided_buffer(char const*) const'
> >> /rados_install/lib/libradosstriper.so: undefined reference to
> >> `ceph::buffer::list::append(ceph::buffer::ptr&&)'
> >> /rados_install/lib/libradosstriper.so: un

[ceph-users] how to build libradosstriper

[Jialin Liu]

Hi,
I'm trying to use the libradosstriper api, but having some trouble in
linking to lradosstriper. I copied only the `required' libraries from an
pre-installed ceph (10.2.10), and put them under my local directory
/rados_install/lib and rados_install/include, on a linux machine.

*/rados_install/lib:*

> libboost_date_time.so.1.53.0  libboost_system.so.1.53.0
> libboost_filesystem-mt.so.1.53.0  libboost_thread-mt.so.1.53.0
> libboost_filesystem.so.1.53.0 libcephfs.so.1.0.0
> libboost_iostreams-mt.so.1.53.0
> libboost_iostreams.so.1.53.0
> libboost_random-mt.so.1.53.0  librados.so.2.0.0
> libboost_regex.so.1.53.0  librados_tp.so.2.0.0
> libboost_system-mt.so.1.53.0  libradosstriper.so.1.0.0
> libcephfs.so   librados.so  libradosstriper.so


*/rados_install/include: *

> |-- rados
> |   |-- buffer.h -> ../buffer.h
> |   |-- buffer_fwd.h -> ../buffer_fwd.h
> |   |-- crc32c.h -> ../crc32c.h
> |   |-- inline_memory.h -> ../inline_memory.h
> |   |-- librados.h
> |   |-- librados.hpp
> |   |-- librgw.h
> |   |-- memory.h -> ../memory.h
> |   |-- objclass.h
> |   |-- page.h -> ../page.h
> |   |-- rados_types.h
> |   |-- rados_types.hpp
> |   `-- rgw_file.h
> `-- radosstriper
> |-- libradosstriper.h
> `-- libradosstriper.hpp


The test code is simple, as attached, and I compiled it with dynamic
linking:

> cc -L/rados_install/lib -I/rados_install/include -o librados_test
> librados_test.c -lrados -lradosstriper


But got the following message, can anyone plz advise?


> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::ptr::ptr(ceph::buffer::ptr&&)'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::ptr::operator=(ceph::buffer::ptr&&)'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `librados::IoCtx::remove(std::string const&, int)'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned int)'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::list::is_provided_buffer(char const*) const'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::list::append(ceph::buffer::ptr&&)'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::list::iterator::iterator(ceph::buffer::list*, unsigned int,
> std::_List_iterator, unsigned int)'
> /rados_install/lib/libradosstriper.so: undefined reference to
> `ceph::buffer::list::list(ceph::buffer::list&&)'


Best,
Jialin
NERSC


librados_test.c
Description: Binary data
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [openstack-dev] container name in swift

[Jialin Liu]

Thanks Iury and John.

Best,
Jialin

On Mon, Apr 2, 2018 at 1:17 PM, Iury Gregory <iurygreg...@gmail.com> wrote:

> According to Swift doc[1]
>
> Length of container names 256 bytes Cannot contain the / character.
> Length of object names 1024 bytes By default, there are no character
> restrictions.
>
> [1] https://docs.openstack.org/swift/latest/api/object_api_
> v1_overview.html
>
>
>
> 2018-04-02 17:00 GMT-03:00 Jialin Liu <jaln...@lbl.gov>:
>
>> Hi John,
>> What is allowed in container name, but not in object name?
>> I need a way to distinguish their name..
>>
>> Best,
>> Jialin
>>
>> On Mon, Apr 2, 2018 at 11:56 AM, John Dickinson <m...@not.mn> wrote:
>>
>>> no
>>>
>>> On 2 Apr 2018, at 11:46, Jialin Liu wrote:
>>>
>>> > Hi,
>>> > Can a container name in openstack swift contains / ?
>>> > e.g.,
>>> > abc/111/mycontainer
>>> >
>>> >
>>> > Best,
>>> > Jialin
>>> > 
>>> __
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
>
>
> *Att[]'sIury Gregory Melo Ferreira *
> *MSc in Computer Science at UFCG*
>
> *Part of the puppet-manager-core team in OpenStack*
> *Social*: https://www.linkedin.com/in/iurygregory
> *E-mail:  iurygreg...@gmail.com <iurygreg...@gmail.com>*
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] container name in swift

[Jialin Liu]

Hi John,
What is allowed in container name, but not in object name?
I need a way to distinguish their name..

Best,
Jialin

On Mon, Apr 2, 2018 at 11:56 AM, John Dickinson <m...@not.mn> wrote:

> no
>
> On 2 Apr 2018, at 11:46, Jialin Liu wrote:
>
> > Hi,
> > Can a container name in openstack swift contains / ?
> > e.g.,
> > abc/111/mycontainer
> >
> >
> > Best,
> > Jialin
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] container name in swift

[Jialin Liu]

Hi,
Can a container name in openstack swift contains / ?
e.g.,
abc/111/mycontainer


Best,
Jialin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Computer-go] FDG18 - Call for Competitions

[Jialin Liu]

[Apologies for cross posting.]

The Foundations of Digital Games 2018 (FDG18) Organizing Committee invites 
proposals for new competitions. Proposals are due by February 28, 2018, and 
will be reviewed based on their relevance to the FDG community. The 
competitions involve well-known games, defining a set of rules and objectives 
for determining the score of each player. Some examples of competition are:
the General Video Game AI competitions
Angry Birds Level Generation
Computer Game Olympiads (including Chess, Amazons, Backgammon, Bridge, Chinese 
Chess, Dots and Boxes, Draughts, Go, LOA, Shogi, …) 
Dota2 Bot
Fighting Game AI
Game Data Mining
General Video Game AI
Geometry Friends Cooperative Game AI
microRTS AI
Ms. Pac-Man Vs Ghost Team
Showdown AI
StarCraft AI
Text-Based Adventure AI
Visual Doom AI.
Please, contact the competition chair (jialin@qmul.ac.uk 
) to send your proposal. The final decision will 
be made before March 5, 2018.

Please, include the following information with your proposal:
Organizers’ names.
Competition title.
Description of the competition (no more than 200 words).
Web address.

The following competitions have already been confirmed, and will hosted be at 
FDG 2018:

The Two-player General Video Game AI Competiton
Organizers: Raluca D. Gaina, Diego Perez Liebana, Simon M. Lucas
Description: The GVG-AI Competition explores the problem of creating 
controllers for general video game playing. How would you create a single agent 
that is able to play any game it is given? Could you program an agent that is 
able to play a wide variety of games, without knowing which games are to be 
played? The Two-Player track brings the additional challenge of competing or 
cooperating with another intelligent agent. This track will count 50 two-player 
games for training, 10 for validation and 10 for test to compute the final 
rankings.
Web address: www.gvgai.net 

Settlement Generation in Minecraft Competition.
Organizers: To be announced
Description: To be announced

Important deadlines for competition proposals:
Submissions: February 28, 2018
Notifications: March 5, 2018

Check all the details here: http://fdg2018.org/ 

Follow us on Twitter #FDG18 |  @FDGconf ‏

___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Numpy-discussion] Extending C with Python

[Jialin Liu]

With PyObject_CallMethod(pInstance, method_name, "O", py_dims);

Can I pass in a reference and modify its content in python?

Best,
Jialin

On Sat, Feb 3, 2018 at 10:29 AM, Jialin Liu <jaln...@lbl.gov> wrote:

> Thank you guys.
>
> Best,
> Jialin
>
> Sent from my iPhone
>
> On Feb 3, 2018, at 8:11 AM, Sylvain Corlay <sylvain.cor...@gmail.com>
> wrote:
>
> You can also check out pybind11, xtensor, and xtensor-python
>
> The latter enables a high-level numpy-like API on the C++ side.
>
> You can check out the numpy to xtensor cheat sheet:
>
> http://xtensor.readthedocs.io/en/latest/numpy.html
>
> Best,
>
> Sylvain
>
>
> On Thu, Feb 1, 2018 at 12:11 AM, Stefan Seefeld <ste...@seefeld.name>
> wrote:
>
>> On 31.01.2018 17:58, Chris Barker wrote:
>>
>> I'm guessing you could use Cython to make this easier.
>>
>>
>> ... or Boost.Python (http://boostorg.github.io/python), which has
>> built-in support for NumPy (http://boostorg.github.io/pyt
>> hon/doc/html/numpy/index.html), and supports both directions: extending
>> Python with C++, as well as embedding Python into C++ applications.
>>
>>
>> 
>>
>> --
>>
>>   ...ich hab' noch einen Koffer in Berlin...
>>
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extending C with Python

[Jialin Liu]

Thank you guys.

Best,
Jialin 

Sent from my iPhone

> On Feb 3, 2018, at 8:11 AM, Sylvain Corlay  wrote:
> 
> You can also check out pybind11, xtensor, and xtensor-python
> 
> The latter enables a high-level numpy-like API on the C++ side.
> 
> You can check out the numpy to xtensor cheat sheet:
> 
> http://xtensor.readthedocs.io/en/latest/numpy.html
> 
> Best,
> 
> Sylvain
> 
> 
>> On Thu, Feb 1, 2018 at 12:11 AM, Stefan Seefeld  wrote:
>>> On 31.01.2018 17:58, Chris Barker wrote:
>>> I'm guessing you could use Cython to make this easier.
>> 
>> ... or Boost.Python (http://boostorg.github.io/python), which has built-in 
>> support for NumPy 
>> (http://boostorg.github.io/python/doc/html/numpy/index.html), and supports 
>> both directions: extending Python with C++, as well as embedding Python into 
>> C++ applications.
>> 
>> 
>> 
>> -- 
>> 
>>   ...ich hab' noch einen Koffer in Berlin...
>> 
>> 
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Extending C with Python

[Jialin Liu]

Amazing! It works! Thank you Robert.

I've been stuck with this many days.

Best,
Jialin
LBNL/NERSC

On Tue, Jan 30, 2018 at 10:52 PM, Robert Kern <robert.k...@gmail.com> wrote:

> On Wed, Jan 31, 2018 at 3:25 PM, Jialin Liu <jaln...@lbl.gov> wrote:
>
>> Hello,
>> I'm extending C with python (which is opposite way of what people usually
>> do, extending python with C), I'm currently stuck in passing a C array to
>> python layer, could anyone plz advise?
>>
>> I have a C buffer in my C code and want to pass it to a python function.
>> In the C code, I have:
>>
>> npy_intp  dims [2];
>>> dims[0] = 10;
>>> dims[1] = 20;
>>> import_array();
>>> npy_intp m=2;
>>> PyObject * py_dims = PyArray_SimpleNewFromData(1, , NPY_INT16 ,(void
>>> *)dims ); // I also tried NPY_INT
>>> PyObject_CallMethod(pInstance, method_name, "O", py_dims);
>>
>>
>> In the Python code, I want to just print that array:
>>
>> def f(self, dims):
>>
>>print ("np array:%d,%d"%(dims[0],dims[1]))
>>
>>
>>
>> But it only prints the first number correctly, i.e., dims[0]. The second
>> number is always 0.
>>
>
> The correct typecode would be NPY_INTP.
>
> --
> Robert Kern
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Extending C with Python

[Jialin Liu]

Hello,
I'm extending C with python (which is opposite way of what people usually
do, extending python with C), I'm currently stuck in passing a C array to
python layer, could anyone plz advise?

I have a C buffer in my C code and want to pass it to a python function. In
the C code, I have:

npy_intp  dims [2];
> dims[0] = 10;
> dims[1] = 20;
> import_array();
> npy_intp m=2;
> PyObject * py_dims = PyArray_SimpleNewFromData(1, , NPY_INT16 ,(void
> *)dims ); // I also tried NPY_INT
> PyObject_CallMethod(pInstance, method_name, "O", py_dims);


In the Python code, I want to just print that array:

def f(self, dims):

   print ("np array:%d,%d"%(dims[0],dims[1]))



But it only prints the first number correctly, i.e., dims[0]. The second
number is always 0.


Best,
Jialin
LBNL/NERSC
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Computer-go] Final CFP: IEEE Transactions on Games: Special Issue on "Game Competition Frameworks for Research and Education"

[Jialin Liu]

[Apologises for multiple postings]

Call for Papers - IEEE Transactions on Games (ToG)
Special Issue on Game Competition Frameworks for Research and Education

Editors: Jialin Liu, Diego Perez-Liebana, Tristan Cazenave, Ruck Thawonmas


Submission Deadline: 8 February 2018

* IEEE Transactions on Games (ToG) is the former IEEE Transactions on 
Computational Intelligence and AI in Games (TCIAIG).


Games are an ideal domain to study computational intelligence methods because 
they provide affordable, competitive, dynamic, reproducible environments 
suitable for testing new search algorithms, pattern-based evaluation methods, 
or machine learning concepts.Diverse game competitions have been designed for 
different research purposes and some of them have been successfully organised 
for 10 years, such as the game Go competition series and PacMan competition 
series. The past game competitions organised in conferences, industry or as 
private leagues have covered various games, from single-player board/video 
games to real-time strategy games. In different competitions, the participants 
are invited to submit an agent to play a specific game or a set of unknown 
games without intervention of human at least as good as professional human 
players, or to submit an agent to design a game or game rules. These have not 
only received submissions from academic institutions, but also attracted the 
attention of the games industry. Dozens of universities have used different 
game competition frameworks in modules of Game Design, Artificial Intelligence 
or Machine Learning. 

The following is a list of suggested, not exclusive, competitions for this 
special issue:
Angry Birds Level Generation
Computer Game Olympiads (including Chess, Amazons, Backgammon, Bridge, Chinese 
Chess, Dots and Boxes, Draughts, Go, ...)
Dota2 Bot
Fighting Game AI
Game Data Mining
General Video Game AI
Geometry Friends Cooperative Game AI
microRTS AI
Ms. Pac-Man Vs Ghost Team
Showdown AI
StarCraft AI
Text-Based Adventure AI
Visual Doom AI
We invite the submission of papers about high quality work on game competition 
frameworks, entry submissions, their use as research testbeds to obtain novel 
experimental results, or as educational and teaching material. Regular, short 
and letter papers are invited to this special issue, with the following 
suggestion for these lengths: 
Letter papers detailing use of competitions as educational or teaching material 
OR describing competition entries; 
Short papers with a technical description of the game competition framework 
(including link to the released code of the benchmark) OR a description of 
competition entries; 
Regular papers describing work using a competition benchmark as a research 
environment for novel experimental results, OR description  of the game 
competition including analysis of the top entries and final results.
Competition organisers and participants are encouraged to communicate and 
collaborate with each other to avoid duplicating descriptions of framework, 
rules, entries, etc. For more information, see the special issue webpage 
<http://gameai.eecs.qmul.ac.uk/si-tog/>.

Authors should follow normal TOG guidelines for their submissions, but clearly 
identify their papers for this special issue during the submission process. 
Extended versions of previously published conference or workshop papers are 
welcome, provided that the journal paper is a significant extension, and is 
accompanied by a cover letter explaining the additional contribution. See here 
<http://cis.ieee.org/ieee-transactions-on-computational-intelligence-and-ai-in-games.html>
 for author information and page length limit.___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[Computer-go] Final CFP: IEEE WCCI/CEC Special Session - Computational Intelligence for Games

[Jialin Liu]

[Apologises for multiple postings]

Call for Papers: Special Session on Computational Intelligence for Games
IEEE WCCI / CEC 2018 <http://www.ecomp.poli.br/~wcci2018/> - Rio de Janeiro, 
Brazil, 8-13 July 2018
Chairs: Daniel Ashlock, Jialin Liu, Santiago Ontañón
Webpages: http://www.ecomp.poli.br/~wcci2018/cross-disciplinary-sessions/#cdss8 
<http://www.ecomp.poli.br/~wcci2018/cross-disciplinary-sessions/#cdss8> and 
http://gameai.eecs.qmul.ac.uk/sscig-wcci2018/ 
<http://gameai.eecs.qmul.ac.uk/sscig-wcci2018/>


Important Dates
Paper submission: 1 February 2018 (extended deadline)
Notification: 15 March 2018

Please select “CDSS-08: Computational Intelligence for Games” as topic when 
making a submission via http://ieee-cis.org/conferences/cec2018/upload.php 
<http://ieee-cis.org/conferences/cec2018/upload.php>.

Corresponding contact: Daniel Ashlock <dashl...@uoguelph.ca 
<mailto:dashl...@uoguelph.ca>>

Aim
Games are an ideal domain to study computational intelligence (CI) methods 
because they provide affordable, competitive, dynamic, reproducible 
environments suitable for testing new search algorithms, pattern-based 
evaluation methods, or learning concepts.  Games scale from simple problems for 
developing algorithms to incredibly hard problems for testing algorithms to the 
limit.  They are also interesting to observe, fun to play, and very attractive 
to students. Additionally, there is great potential for CI methods to improve 
the design and development of both computer games as well as tabletop games, 
board games, and puzzles.  This special session aims at gathering leaders and 
neophytes in games research as well as practitioners in this field who research 
applications of computational intelligence methods to computer games.

Topics
In general, papers are welcome that consider all kinds of applications of 
methods (evolutionary computation, supervised learning, neural learning, 
unsupervised learning, fuzzy systems, game-tree search, rolling horizon 
algorithms, MCTS, etc.) to games (card games, board games, mathematical games, 
action games, strategy games, role-playing games, arcade games, serious games, 
etc.). 

Examples include but are not limited to
Adaptation in games
Automatic game testing
Novel games that use CI techniques
Coevolution in games
Comparative studies (e.g. CI versus human-designed players)
Dynamic difficulty in games
Games as test-beds for algorithms
Imitating human players
Learning to play games
Multi-agent and multi-strategy learning
Player/opponent modelling
Procedural content generation
CI for Serious Games (e.g., games for health care, education or training)
Results of game-based CI and open competitions

About the organisers
Daniel Ashlock is a Professor of Mathematics at the University of Guelph.  He 
is a member of the IEEE Computational
Intelligence Societies’ Games Technical Committee, serves as an associate 
editor of both the IEEE Transactions on Games and the
new journal Game and Puzzle Design.  Dr. Ashlock is a life-long referee of 
roleplaying games and loves the problems that arise in game and puzzle design.  
Dr. Ashlock is the author of 270 peer reviewed scientific publications, about a 
third of which are on the topic of games, including mathematical games, 
procedural content generation, and puzzle design.  Across his work the issue of 
representation, the study of how to phrase problems for the computer, appears 
in a starring role.  Dr. Ashlock is the Chief of Innovation at Ashlock and 
McGuinness Consulting, Inc.  He teaches and does research in Guelph, Ontario, 
Canada.

Jialin Liu is a Postdoctoral Research Associate at Queen Mary University of 
London (QMUL, UK) since the creation of the Game AI research group in August 
2017. Before joining QMUL, she was a Senior Research Officer at the Games and 
AI research group of University of Essex (UK) between March 2016 and July 2017. 
 Jialin received her PhD in Computer Science from the University Paris-Saclay & 
Inria Saclay (France) in December 2015. Her research interests include general 
game playing, robust game playing, automatic game design, black-box noisy 
optimisation and portfolio algorithms. Jialin is chairing the IEEE CIS Student 
Games-Based Competitions Sub-Committee and also serves in the Games Technical 
Committee, Webinars Sub-Committee, Women in Computational Intelligence 
Sub-Committee and Young Professionals Sub-Committee. She serves as Program 
Co-Chair at IEEE CIG18.

Santiago Ontañón is an Associate Professor in the Computer Science Department 
at Drexel University. His main research interests are game AI, case-based 
reasoning and machine learning, fields in which he has published more than 150 
peer reviewed papers. He obtained his PhD from the Autonomous University of 
Barcelona (UAB), Spain in 2005. Before joining Drexel University, he held 
postdoctoral research positions at the Artificial Intelligence Research 
Institute (I

[Computer-go] [Deadline extended] CFP: IEEE WCCI-CEC Special Session - Computational Intelligence for Games

[Jialin Liu]

[Apologises for multiple postings]

Call for Papers: Special Session on Computational Intelligence for Games
IEEE WCCI / CEC 2018 <http://www.ecomp.poli.br/~wcci2018/> - Rio de Janeiro, 
Brazil, 8-13 July 2018
Chairs: Daniel Ashlock, Jialin Liu, Santiago Ontañón
Webpage: http://gameai.eecs.qmul.ac.uk/sscig-wcci2018/ 
<http://gameai.eecs.qmul.ac.uk/sscig-wcci2018/>
Important Dates
Paper submission: extended to 1 February 2018
Notification: 15 March 2018

Please select “CDSS-08: Computational Intelligence for Games” as topic when 
making a submission via http://ieee-cis.org/conferences/cec2018/upload.php 
<http://ieee-cis.org/conferences/cec2018/upload.php>.

Corresponding contact: Daniel Ashlock <dashl...@uoguelph.ca 
<mailto:dashl...@uoguelph.ca>>

Aim
Games are an ideal domain to study computational intelligence (CI) methods 
because they provide affordable, competitive, dynamic, reproducible 
environments suitable for testing new search algorithms, pattern-based 
evaluation methods, or learning concepts.  Games scale from simple problems for 
developing algorithms to incredibly hard problems for testing algorithms to the 
limit.  They are also interesting to observe, fun to play, and very attractive 
to students. Additionally, there is great potential for CI methods to improve 
the design and development of both computer games as well as tabletop games, 
board games, and puzzles.  This special session aims at gathering leaders and 
neophytes in games research as well as practitioners in this field who research 
applications of computational intelligence methods to computer games.

Topics
In general, papers are welcome that consider all kinds of applications of 
methods (evolutionary computation, supervised learning, neural learning, 
unsupervised learning, fuzzy systems, game-tree search, rolling horizon 
algorithms, MCTS, etc.) to games (card games, board games, mathematical games, 
action games, strategy games, role-playing games, arcade games, serious games, 
etc.). 

Examples include but are not limited to
Adaptation in games
Automatic game testing
Novel games that use CI techniques
Coevolution in games
Comparative studies (e.g. CI versus human-designed players)
Dynamic difficulty in games
Games as test-beds for algorithms
Imitating human players
Learning to play games
Multi-agent and multi-strategy learning
Player/opponent modelling
Procedural content generation
CI for Serious Games (e.g., games for health care, education or training)
Results of game-based CI and open competitions

About the organisers
Daniel Ashlock is a Professor of Mathematics at the University of Guelph.  He 
is a member of the IEEE Computational
Intelligence Societies’ Games Technical Committee, serves as an associate 
editor of both the IEEE Transactions on Games and the
new journal Game and Puzzle Design.  Dr. Ashlock is a life-long referee of 
roleplaying games and loves the problems that arise in game and puzzle design.  
Dr. Ashlock is the author of 270 peer reviewed scientific publications, about a 
third of which are on the topic of games, including mathematical games, 
procedural content generation, and puzzle design.  Across his work the issue of 
representation, the study of how to phrase problems for the computer, appears 
in a starring role.  Dr. Ashlock is the Chief of Innovation at Ashlock and 
McGuinness Consulting, Inc.  He teaches and does research in Guelph, Ontario, 
Canada.

Jialin Liu is a Postdoctoral Research Associate at Queen Mary University of 
London (QMUL, UK) since the creation of the Game AI research group in August 
2017. Before joining QMUL, she was a Senior Research Officer at the Games and 
AI research group of University of Essex (UK) between March 2016 and July 2017. 
 Jialin received her PhD in Computer Science from the University Paris-Saclay & 
Inria Saclay (France) in December 2015. Her research interests include general 
game playing, robust game playing, automatic game design, black-box noisy 
optimisation and portfolio algorithms. Jialin is chairing the IEEE CIS Student 
Games-Based Competitions Sub-Committee and also serves in the Games Technical 
Committee, Webinars Sub-Committee, Women in Computational Intelligence 
Sub-Committee and Young Professionals Sub-Committee. She serves as Program 
Co-Chair at IEEE CIG18.

Santiago Ontañón is an Associate Professor in the Computer Science Department 
at Drexel University. His main research interests are game AI, case-based 
reasoning and machine learning, fields in which he has published more than 150 
peer reviewed papers. He obtained his PhD from the Autonomous University of 
Barcelona (UAB), Spain in 2005. Before joining Drexel University, he held 
postdoctoral research positions at the Artificial Intelligence Research 
Institute (IIIA) in Barcelona, Spain, at the Georgia Institute of Technology 
(GeorgiaTech) in Atlanta, USA, and lectured at the Un

[Computer-go] 3rd CFP: IEEE WCCI/CEC Special Session - Computational Intelligence for Games

[Jialin Liu]

[Apologises for multiple postings]

Call for Papers: Special Session on Computational Intelligence for Games
IEEE WCCI / CEC 2018 <http://www.ecomp.poli.br/~wcci2018/> - Rio de Janeiro, 
Brazil, 8-13 July 2018
Chairs: Daniel Ashlock, Jialin Liu, Santiago Ontañón
Webpage: http://gameai.eecs.qmul.ac.uk/sscig-wcci2018/ 
<http://gameai.eecs.qmul.ac.uk/sscig-wcci2018/>
Important Dates
Paper submission: 15 January 2018
Notification: 15 March 2018

Please select “CDSS-08: Computational Intelligence for Games” as topic when 
making a submission via http://ieee-cis.org/conferences/cec2018/upload.php 
<http://ieee-cis.org/conferences/cec2018/upload.php>.

Corresponding contact: Daniel Ashlock <dashl...@uoguelph.ca 
<mailto:dashl...@uoguelph.ca>>

Aim
Games are an ideal domain to study computational intelligence (CI) methods 
because they provide affordable, competitive, dynamic, reproducible 
environments suitable for testing new search algorithms, pattern-based 
evaluation methods, or learning concepts.  Games scale from simple problems for 
developing algorithms to incredibly hard problems for testing algorithms to the 
limit.  They are also interesting to observe, fun to play, and very attractive 
to students. Additionally, there is great potential for CI methods to improve 
the design and development of both computer games as well as tabletop games, 
board games, and puzzles.  This special session aims at gathering leaders and 
neophytes in games research as well as practitioners in this field who research 
applications of computational intelligence methods to computer games.

Topics
In general, papers are welcome that consider all kinds of applications of 
methods (evolutionary computation, supervised learning, neural learning, 
unsupervised learning, fuzzy systems, game-tree search, rolling horizon 
algorithms, MCTS, etc.) to games (card games, board games, mathematical games, 
action games, strategy games, role-playing games, arcade games, serious games, 
etc.). 

Examples include but are not limited to
Adaptation in games
Automatic game testing
Novel games that use CI techniques
Coevolution in games
Comparative studies (e.g. CI versus human-designed players)
Dynamic difficulty in games
Games as test-beds for algorithms
Imitating human players
Learning to play games
Multi-agent and multi-strategy learning
Player/opponent modelling
Procedural content generation
CI for Serious Games (e.g., games for health care, education or training)
Results of game-based CI and open competitions

About the organisers
Daniel Ashlock is a Professor of Mathematics at the University of Guelph.  He 
is a member of the IEEE Computational
Intelligence Societies’ Games Technical Committee, serves as an associate 
editor of both the IEEE Transactions on Games and the
new journal Game and Puzzle Design.  Dr. Ashlock is a life-long referee of 
roleplaying games and loves the problems that arise in game and puzzle design.  
Dr. Ashlock is the author of 270 peer reviewed scientific publications, about a 
third of which are on the topic of games, including mathematical games, 
procedural content generation, and puzzle design.  Across his work the issue of 
representation, the study of how to phrase problems for the computer, appears 
in a starring role.  Dr. Ashlock is the Chief of Innovation at Ashlock and 
McGuinness Consulting, Inc.  He teaches and does research in Guelph, Ontario, 
Canada.

Jialin Liu is a Postdoctoral Research Associate at Queen Mary University of 
London (QMUL, UK) since the creation of the Game AI research group in August 
2017. Before joining QMUL, she was a Senior Research Officer at the Games and 
AI research group of University of Essex (UK) between March 2016 and July 2017. 
 Jialin received her PhD in Computer Science from the University Paris-Saclay & 
Inria Saclay (France) in December 2015. Her research interests include general 
game playing, robust game playing, automatic game design, black-box noisy 
optimisation and portfolio algorithms. Jialin is chairing the IEEE CIS Student 
Games-Based Competitions Sub-Committee and also serves in the Games Technical 
Committee, Webinars Sub-Committee, Women in Computational Intelligence 
Sub-Committee and Young Professionals Sub-Committee. She serves as Program 
Co-Chair at IEEE CIG18.

Santiago Ontañón is an Associate Professor in the Computer Science Department 
at Drexel University. His main research interests are game AI, case-based 
reasoning and machine learning, fields in which he has published more than 150 
peer reviewed papers. He obtained his PhD from the Autonomous University of 
Barcelona (UAB), Spain in 2005. Before joining Drexel University, he held 
postdoctoral research positions at the Artificial Intelligence Research 
Institute (IIIA) in Barcelona, Spain, at the Georgia Institute of Technology 
(GeorgiaTech) in Atlanta, USA, and lectured at the University of Barcelona 
(UB), Spain._

[openstack-dev] Can swift leverage os page buffer?

[Jialin Liu]

Hi,
I'm new to openstack swift, I'm a HPC user, by several days of exploration
of swift, I have some naive questions:
1. Can swift, e.g., PUT, leverage OS' page buffer?; Is there a Linux kernel
module for swift? What are the existing optimizations for caching the write
in order to gain better bandwidth on spin disks?
2. Does swift support asynchronous PUT/Get?


Also please let me know if the dev list is good for me to ask this kind of
questions.

Best,
Jialin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] leverage page cache in openstack swift

[Jialin Liu]

Hi,
Is there any existing work that leveraging operating system's page cache
for swift?
like many other parallel file systems, lustre, the IO is cached in buffer
and call is returned immediately to user space.

Best,
Jialin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] how to write python object in openstack swift

[Jialin Liu]

Hi John,
That indeed helped. But I'm stuck in using the python interface to make it
work:

_opts = {'object_uu_threads': 10, 'segment_size': 1048576}

 with SwiftService(options=_opts) as swift:

Seems not working.

Also my another question is about the relationship with uu_threads and
segment, my understanding is that if we upload by segments, we should use
more than 1 threads so that parallelism is possible, right?


Best,

Jialin

On Thu, Oct 5, 2017 at 11:32 AM, John Dickinson <m...@not.mn> wrote:

> I'm not familiar with that format, but in general, if you're dealing with
> large objects, you can get better performance by splitting the data in the
> client, uploading each separate segment concurrently, and then creating a
> large object manifest to tie the segments together (allowing future access
> by either segments or as a logical whole).
>
> https://docs.openstack.org/swift/latest/overview_large_
> objects.html#module-swift.common.middleware.slo
>
> --John
>
>
>
> On 5 Oct 2017, at 11:24, Jialin Liu wrote:
>
> Thank you John,
>
> I think I figured out the way. My case is a little bit rare as I'm dealing
> with the HDF5 file format, I can not think of any way to retrieve the
> entire in memory file as a single python object and then serialize it.
> What I did is to use the hdf5.get_file_image function to get a memory
> image of the file, and then use, io.ByteIO(image) to make it a file-like
> object.
>
> So far, it works well, but I believe it is not the best way in terms of
> performance.
>
> Best,
> Jialin
>
> On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson <m...@not.mn> wrote:
>
>> If you've got an arbitrary object in Python, you'll need to serialize it
>> to a file-like object. You could keep it in memory and use a StringIO
>> type, or you could serialize it to disk and open() it like any other
>> file.
>>
>> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
>> are. You, as the Swift client (i.e. API user), need to dump those bytes on
>> the network to send them to Swift. As long as you're transforming your
>> Python object[s] in some regular way that makes sense in your application,
>> it doesn't matter what bytes you send to Swift.
>>
>> --John
>>
>> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>>
>> Hi,
>> It seems to me that openstack swift only supports file upload/download, is
>> it possible to put a python object to swift store?
>> The doc says we could use file-like object, e.g., StringIO, but this is
>> very limited. I'd like to write a numpy array or other python object into
>> the swift store, can anybody tell me the solution?
>>
>> Best,
>> Jialin
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] how to write python object in openstack swift

[Jialin Liu]

Thank you John,

I think I figured out the way. My case is a little bit rare as I'm dealing
with the HDF5 file format, I can not think of any way to retrieve the
entire in memory file as a single python object and then serialize it.
What I did is to use the hdf5.get_file_image function to get a memory image
of the file, and then use, io.ByteIO(image) to make it a file-like object.

So far, it works well, but I believe it is not the best way in terms of
performance.

Best,
Jialin

On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson <m...@not.mn> wrote:

> If you've got an arbitrary object in Python, you'll need to serialize it
> to a file-like object. You could keep it in memory and use a StringIO
> type, or you could serialize it to disk and open() it like any other file.
>
> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
> are. You, as the Swift client (i.e. API user), need to dump those bytes on
> the network to send them to Swift. As long as you're transforming your
> Python object[s] in some regular way that makes sense in your application,
> it doesn't matter what bytes you send to Swift.
>
> --John
>
> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>
> Hi,
> It seems to me that openstack swift only supports file upload/download, is
> it possible to put a python object to swift store?
> The doc says we could use file-like object, e.g., StringIO, but this is
> very limited. I'd like to write a numpy array or other python object into
> the swift store, can anybody tell me the solution?
>
> Best,
> Jialin
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] how to write python object in openstack swift

[Jialin Liu]

Hi,
It seems to me that openstack swift only supports file upload/download, is
it possible to put a python object to swift store?
The doc says we could use file-like object, e.g., StringIO, but this is
very limited. I'd like to write a numpy array or other python object into
the swift store, can anybody tell me the solution?

Best,
Jialin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Computer-go] 1st CFP: IEEE Transactions on Games (ToG): Special Issue on "Game Competition Frameworks for Research and Education"

[Jialin Liu]

[Apologises for multiple postings]

Call for Papers


IEEE Transactions on Games (ToG)
Special Issue on Game Competition Frameworks for Research and Education

Editors: Jialin Liu, Diego Perez-Liebana, Tristan Cazenave, Ruck Thawonmas


Submission Deadline: 8 January 2018


Games are an ideal domain to study computational intelligence methods because 
they provide affordable, competitive, dynamic, reproducible environments 
suitable for testing new search algorithms, pattern-based evaluation methods, 
or machine learning concepts.Diverse game competitions have been designed for 
different research purposes and some of them have been successfully organised 
for 10 years, such as the game Go competition series and PacMan competition 
series. The past game competitions organised in conferences, industry or as 
private leagues have covered various games, from single-player board/video 
games to real-time strategy games. In different competitions, the participants 
are invited to submit an agent to play a specific game or a set of unknown 
games without intervention of human at least as good as professional human 
players, or to submit an agent to design a game or game rules. These have not 
only received submissions from academic institutions, but also attracted the 
attention of the games industry. Dozens of universities have used different 
game competition frameworks in modules of Game Design, Artificial Intelligence 
or Machine Learning. 

The following is a list of suggested, not exclusive, competitions for this 
special issue:
Angry Birds Level Generation
Computer Game Olympiads (including Chess, Amazons, Backgammon, Bridge, Chinese 
Chess, Dots and Boxes, Draughts, Go, ...)
Dota2 Bot
Fighting Game AI
Game Data Mining
General Video Game AI
Geometry Friends Cooperative Game AI
microRTS AI
Ms. Pac-Man Vs Ghost Team
Showdown AI
StarCraft AI
Text-Based Adventure AI
Visual Doom AI
We invite the submission of papers about high quality work on game competition 
frameworks, entry submissions, their use as research testbeds to obtain novel 
experimental results, or as educational and teaching material. Regular, short 
and letter papers are invited to this special issue, with the following 
suggestion for these lengths: 
Letter papers detailing use of competitions as educational or teaching material 
OR describing competition entries; 
Short papers with a technical description of the game competition framework 
(including link to the released code of the benchmark) OR a description of 
competition entries; 
Regular papers describing work using a competition benchmark as a research 
environment for novel experimental results, OR description  of the game 
competition including analysis of the top entries and final results.
Competition organisers and participants are encouraged to communicate and 
collaborate with each other to avoid duplicating descriptions of framework, 
rules, entries, etc. For more information, see the special issue webpage 
<http://gameai.eecs.qmul.ac.uk/si-tog/>.

Authors should follow normal TOG guidelines for their submissions, but clearly 
identify their papers for this special issue during the submission process. 
Extended versions of previously published conference or workshop papers are 
welcome, provided that the journal paper is a significant extension, and is 
accompanied by a cover letter explaining the additional contribution. See here 
<http://cis.ieee.org/ieee-transactions-on-computational-intelligence-and-ai-in-games.html>
 for author information and page length limit.

* From 1st January 2018, the journal IEEE Transactions on Computational 
Intelligence and AI in Games (TCIAIG) will be called IEEE Transactions on Games 
(ToG).___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

[UAI] Call for papers: Special Session on Computational Intelligence and Games (CIG) at IEEE CEC2017

[Jialin Liu]

** Apologies for multiple postings **
** Please kindly forward to those who may be interested **
-
CALL FOR PAPERS
*Special Session on Computational Intelligence and Games (CIG) at IEEE
Congress on Evolutionary Computation (IEEE CEC) 2017*

*IEEE CEC 2017 - Special Session*
Donostia - San Sebasti*á*n, Spa*i*n
June 5-8, 2017

*Special Session Website: http://www.liujialin.tech/sscig.html
<http://www.liujialin.tech/sscig.html>*

SPECIAL SESSION
Games are an ideal domain to study computational intelligence (CI) methods
because they provide affordable, competitive, dynamic, reproducible
environments suitable for testing new search algorithms, pattern-based
evaluation methods, or learning concepts. They are also interesting to
observe, fun to play, and very attractive to students. Additionally, there
is great potential for CI methods to improve the design and development of
both computer games and non-digital games such as board games. This special
session aims at gathering not only leading researchers, but also young
researchers as well as practitioners in this field who research
applications of computational intelligence methods to computer games.

TOPICS OF INTEREST
In general, papers are welcome that consider all kinds of applications of
CI methods (evolutionary computation, supervised learning, unsupervised
learning, fuzzy systems, game-tree search, rolling horizon algorithms,
MCTS, etc.) to games (card games, board games, mathematical games, action
games, strategy games, role-playing games, arcade games, serious games,
etc.). Examples include
Adaptation in games
Automatic game testing
Coevolution in games
Comparative studies (e.g. CI versus human-designed players)
Dynamic difficulty in games
Games as test-beds for CI algorithms
Imitating human players
Learning to play games
Multi-agent and multi-strategy learning
Player/opponent modelling
Procedural content generation
Results of game-based CI competitions
Results of open competitions

IMPORTANT DATES
Paper Submission : January 16, 2017
Notification : February 26, 2017
Final Paper Submission : March 12, 2017
Special Session : June 5-8, 2017

SUBMISSION GUIDELINES

*Manuscripts should be prepared according to the standard format and page
limit of regular papers specified in CEC'2017.*
Special session papers should be uploaded online through the paper
submission website of IEEE CEC 2017 by January 16, 2017.
Please select the corresponding special session name ("Computational
Intelligence and Games") as the “main research topic” in submission.
*Special session papers will be treated in the same way as regular papers
and included in the conference proceedings.*
For the latest information on important dates, the template and details,
please visit http://www.cec2017.org.

ORGANIZERS

Daniel Ashlock, dashl...@uoguelph.ca
Dept. of Mathematics and Statistics, University of Guelph, Canada

Antonio J. Fernández-Leiva, af...@lcc.uma.es
School of Computer Studies, University of Málaga, Span

Jialin Liu, jialin@essex.ac.uk
 School of Computer Science and Electronic Engineering, University of
Essex, UK



Dr Jialin Liu PhD
Senior Research Officer
CSEE, University of Essex, UK

E jialin@essex.ac.uk
► <http://www.essex.ac.uk/centres/psycho/>http://www.liujialin.tech/

"Truth is ever to be found in simplicity, and not in the multiplicity and
confusion of things.” — Isaac Newton
___
uai mailing list
uai@ENGR.ORST.EDU
https://secure.engr.oregonstate.edu/mailman/listinfo/uai

Re: [Hdf-forum] h5copy, segmentation fault

[Jialin Liu]

Hi, It turns out that the data is highly compressed, once loaded in memory, it 
consumes 100G or more memory which exceeds the single node RAM capacity. 
I think that maybe the reason, Thanks. 


Best,
Jialin

> On Jul 14, 2016, at 2:09 PM, Jialin Liu <jaln...@lbl.gov> wrote:
> 
> Hi, 
> I’m using the h5copy to convert an netcdf file to hdf5, it returns 
> segmentation fault, the file size is 24G, 
> the specific dataset that i tried to convert is H5T_STD_I16LE and ( 1617, 
> 4320, 8640).
> 
> Any idea about this error? Thanks, 
> 
> Best,
> Jialin

___
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

[Hdf-forum] h5copy, segmentation fault

[Jialin Liu]

Hi, 
I’m using the h5copy to convert an netcdf file to hdf5, it returns segmentation 
fault, the file size is 24G, 
the specific dataset that i tried to convert is H5T_STD_I16LE and ( 1617, 4320, 
8640).

Any idea about this error? Thanks, 

Best,
Jialin___
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Re: Is storage resources counted during the scheduling

[Jialin Liu]

Thanks Ted, 
but that page seems to be scheduling policy, I have no idea of what resources 
are considered in the scheduling. 

And for scheduling, I’m wondering, in case of just one application, is there 
also a scheduling process?
otherwise, why I see some launching delay in the tasks. (well, this might be 
another question). Thanks. 

Best,
Jialin
> On Apr 11, 2016, at 3:18 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> 
> See 
> https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application
>  
> <https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application>
> 
> On Mon, Apr 11, 2016 at 3:15 PM, Jialin Liu <jaln...@lbl.gov 
> <mailto:jaln...@lbl.gov>> wrote:
> Hi Spark users/experts,
> 
> I’m wondering how does the Spark scheduler work?
> What kind of resources will be considered during the scheduling, does it 
> include the disk resources or I/O resources, e.g., number of IO ports.
> Is network resources considered in that?
> 
> My understanding is that only CPU is considered, right?
> 
> Best,
> Jialin
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org 
> <mailto:user-unsubscr...@spark.apache.org>
> For additional commands, e-mail: user-h...@spark.apache.org 
> <mailto:user-h...@spark.apache.org>
> 
> 

Is storage resources counted during the scheduling

[Jialin Liu]

Hi Spark users/experts, 

I’m wondering how does the Spark scheduler work?
What kind of resources will be considered during the scheduling, does it 
include the disk resources or I/O resources, e.g., number of IO ports. 
Is network resources considered in that?

My understanding is that only CPU is considered, right?

Best,
Jialin
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: spark launching range is 10 mins

[Jialin Liu]

Hi,
I have set the partitions as 6000, and requested 100 nodes, with 32
cores each node,
and the number of executors is 32 per node

spark-submit --master $SPARKURL --executor-cores 32 --driver-memory
20G --executor-memory 80G single-file-test.py


And I'm reading a 2.2 TB, the code, just has simple two steps,
rdd=sc.read
rdd.count
Then I checked the log file, and history server, it shows that the
count stage has a really large tasks launching range, e.g.,

16/03/19 22:40:17
16/03/19 22:30:56

which is about 10 minutes,
Has anyone experienced this before?
Could you please let me know the reason and internal of Spark relating
to this issue,
and how to resolve it? Thanks much.

Best,
Jialin

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

[Hdf-forum] unable to use hdf5-java

[Jialin Liu]

Hi,

I want to use the hdf5 java library on mac, but always got this error:
I always got this error:

java.lang.UnsatisfiedLinkError: no jhdf5 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
…

The test program I used is H5Ex_D_Hyperslab.java
which is downloaded online. 

I followed the steps in 
https://www.hdfgroup.org/products/java/hdf-object/index.html 

Apparently, it’s outdated( Last modified: 15 December 2014), for example, there 
is no jhdf.jar, instead it has been renamed as jarhdf.jar

I have tried all existing solutions online(e.g., adding the pre-built library 
to CLASSPATH), but still couldn’t fix it. 
Could you please give it a try and let me know how to configure it? Thanks. 


Best,
Jialin___
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

[Hdf-forum] login password

[Jialin Liu]

Hi, 
I try login onto the h5serv http://tall.data.hdfgroup.org:7253/ 

but it requires username and password. Where can I have one?

Thanks,
Jialin___
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

a error in git documentation

[Jialin Liu]

In http://git-scm.com/book/en/Git-Basics-Recording-Changes-to-the-Repository

Notice how you don’t have to run git add on the benchmarks.rb file in this case 
before you commit.

I think how should be changed to now--
To unsubscribe from this list: send the line unsubscribe git in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html