Re: Audio file metadata parser?

2025-07-18 Thread Andy Valencia via Digitalmars-d-learn
On Saturday, 19 July 2025 at 00:23:28 UTC, Jonathan M Davis wrote: If you're looking to operate on tags in audio files, then there's https://code.dlang.org/packages/taglib-d, which might do what you need, but I don't know much about it, so I can't say for sure. Thank you! That handles my ne

Re: Audio file metadata parser?

2025-07-18 Thread Jonathan M Davis via Digitalmars-d-learn
On Friday, July 18, 2025 4:47:04 PM Mountain Daylight Time Andy Valencia via Digitalmars-d-learn wrote: > Is anyone aware of a library like "tinytag" (a Python module)? > > https://github.com/devsnd/tinytag > > I guess I'll wrestle with porting it to dlang if there's nothing > available. > > tinyt

Audio file metadata parser?

2025-07-18 Thread Andy Valencia via Digitalmars-d-learn
Is anyone aware of a library like "tinytag" (a Python module)? https://github.com/devsnd/tinytag I guess I'll wrestle with porting it to dlang if there's nothing available. tinytag is nice, because it handles wav/ogg/flac/opus/mp3 uniformly. Thanks, Andy

Re: moving from classical lex/yacc to pegged parser

2024-05-10 Thread Dukc via Digitalmars-d-learn
Dmitry Ponyatov kirjoitti 9.5.2024 klo 11.30: > And I also can't figure out how to inherit `ParseTree` with all my script language objects to get AST right from pegged parser. Should I use some superloop with lot of matches to process parsed `pt` tree into something I need myself, to

moving from classical lex/yacc to pegged parser

2024-05-09 Thread Dmitry Ponyatov via Digitalmars-d-learn
loat value` for numbers map attr; vector nested; } ``` And I also can't figure out how to inherit `ParseTree` with all my script language objects to get AST right from pegged parser. Should I use some superloop with lot of matches to process parsed `pt` tree into something I need

Re: What parser generator can let me run arbitrary code in its match rules?

2023-11-20 Thread Julian Fondren via Digitalmars-d-learn
On Monday, 20 November 2023 at 23:50:24 UTC, Dmitry Ponyatov wrote: - not abandoned years ago - documentation and commented samples presenets - CTFE the best https://code.dlang.org/packages/pegged

Re: What parser generator can let me run arbitrary code in its match rules?

2023-11-20 Thread Guillaume Piolat via Digitalmars-d-learn
#x27; book explain recursive descent parser, including parsing binary operators with priorities.

Re: What parser generator can let me run arbitrary code in its match rules?

2023-11-20 Thread Dmitry Ponyatov via Digitalmars-d-learn
Or maybe someone advice me some set of books deeply targets for learning of binary and symmetric parsing (such as binpac), DCG in C or using generators in D, etc to let me write my own lib. Everething I found (besides the Dragon book) uses black magic with Haskell, Lisp etc.

What parser generator can let me run arbitrary code in its match rules?

2023-11-20 Thread Dmitry Ponyatov via Digitalmars-d-learn
- not abandoned years ago - documentation and commented samples presenets - CTFE the best

Re: Parser

2023-06-15 Thread Ben Jones via Digitalmars-d-learn
On Wednesday, 14 June 2023 at 09:28:57 UTC, Cecil Ward wrote: I’m thinking that I might had to end up writing a partial, rather rough parser for parts of the D language. Could I get some suggestions for help that I might find in the way of software components? D has a very powerful regex

Parser

2023-06-14 Thread Cecil Ward via Digitalmars-d-learn
I’m thinking that I might had to end up writing a partial, rather rough parser for parts of the D language. Could I get some suggestions for help that I might find in the way of software components? D has a very powerful regex module, I believe. I have been writing inline asm library routines

Re: Is there such a JSON parser?

2023-01-02 Thread torhu via Digitalmars-d-learn
On Monday, 2 January 2023 at 21:36:10 UTC, Steven Schveighoffer wrote: On 1/1/23 6:28 PM, torhu wrote: I need to parse some JSON data into various data structures, so I'm looking for a parser based on events or ranges. One that doesn't just load the file and build a data struc

Re: Is there such a JSON parser?

2023-01-02 Thread Steven Schveighoffer via Digitalmars-d-learn
On 1/1/23 6:28 PM, torhu wrote: I need to parse some JSON data into various data structures, so I'm looking for a parser based on events or ranges. One that doesn't just load the file and build a data structure that represents the whole thing. So not std.json, at least. It'

Re: Is there such a JSON parser?

2023-01-02 Thread Chris Piker via Digitalmars-d-learn
On Monday, 2 January 2023 at 14:56:27 UTC, SealabJaster wrote: Are you asking for a SAX-styled parser for JSON? I have an upcoming project (about 3-6 months away) that could make use of this as well. If you need someone to try it out please let me know and I'll give it a spin. Good

Re: Is there such a JSON parser?

2023-01-02 Thread torhu via Digitalmars-d-learn
On Monday, 2 January 2023 at 14:56:27 UTC, SealabJaster wrote: Are you asking for a SAX-styled parser for JSON? Yes, I actually want to replace a SAX parser.

Re: Is there such a JSON parser?

2023-01-02 Thread torhu via Digitalmars-d-learn
On Monday, 2 January 2023 at 05:44:33 UTC, thebluepandabear wrote: You might want to try the following: https://github.com/libmir/asdf I had a look at that, but I think it just loads the whole file into it's own data structure. And then you can use attributes to get it to fill structs with d

Re: Is there such a JSON parser?

2023-01-02 Thread SealabJaster via Digitalmars-d-learn
On Sunday, 1 January 2023 at 23:28:12 UTC, torhu wrote: I need to parse some JSON data into various data structures, so I'm looking for a parser based on events or ranges. One that doesn't just load the file and build a data structure that represents the whole thing. So not std.json

Re: Is there such a JSON parser?

2023-01-01 Thread thebluepandabear via Digitalmars-d-learn
On Sunday, 1 January 2023 at 23:28:12 UTC, torhu wrote: I need to parse some JSON data into various data structures, so I'm looking for a parser based on events or ranges. One that doesn't just load the file and build a data structure that represents the whole thing. So not std.json

Re: Is there such a JSON parser?

2023-01-01 Thread Salih Dincer via Digitalmars-d-learn
On Sunday, 1 January 2023 at 23:28:12 UTC, torhu wrote: I need to parse some JSON data into various data structures, so I'm looking for a parser based on events or ranges. One that doesn't just load the file and build a data structure that represents the whole thing. So not std.json

Is there such a JSON parser?

2023-01-01 Thread torhu via Digitalmars-d-learn
I need to parse some JSON data into various data structures, so I'm looking for a parser based on events or ranges. One that doesn't just load the file and build a data structure that represents the whole thing. So not std.json, at least.

Re: Should a parser type be a struct or class?

2020-06-18 Thread Meta via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 11:50:27 UTC, Per Nordlöw wrote: Should a range-compliant aggregate type realizing a parser be encoded as a struct or class? In dmd `Lexer` and `Parser` are both classes. In general how should I reason about whether an aggregate type should be encoded as a

Re: Should a parser type be a struct or class?

2020-06-18 Thread welkam via Digitalmars-d-learn
Oh an also https://github.com/dlang/dmd/pull/9899

Re: Should a parser type be a struct or class?

2020-06-18 Thread welkam via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 14:32:09 UTC, Adam D. Ruppe wrote: On Wednesday, 17 June 2020 at 14:24:01 UTC, Stefan Koch wrote: Parser in dmd does even inherit from Lexer. why would a parser ever inherit from a lexer? So you can write nextToken() instead of lexer.nextToken()

Re: Should a parser type be a struct or class?

2020-06-17 Thread user1234 via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 11:50:27 UTC, Per Nordlöw wrote: Should a range-compliant aggregate type realizing a parser be encoded as a struct or class? In dmd `Lexer` and `Parser` are both classes. In general how should I reason about whether an aggregate type should be encoded as a

Re: Should a parser type be a struct or class?

2020-06-17 Thread H. S. Teoh via Digitalmars-d-learn
On Wed, Jun 17, 2020 at 02:32:09PM +, Adam D. Ruppe via Digitalmars-d-learn wrote: > On Wednesday, 17 June 2020 at 14:24:01 UTC, Stefan Koch wrote: > > Parser in dmd does even inherit from Lexer. > > why would a parser ever inherit from a lexer? Because, unlike a regula

Re: Should a parser type be a struct or class?

2020-06-17 Thread Adam D. Ruppe via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 14:24:01 UTC, Stefan Koch wrote: Parser in dmd does even inherit from Lexer. why would a parser ever inherit from a lexer?

Re: Should a parser type be a struct or class?

2020-06-17 Thread Stefan Koch via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 11:50:27 UTC, Per Nordlöw wrote: Should a range-compliant aggregate type realizing a parser be encoded as a struct or class? In dmd `Lexer` and `Parser` are both classes. In general how should I reason about whether an aggregate type should be encoded as a

Re: Should a parser type be a struct or class?

2020-06-17 Thread H. S. Teoh via Digitalmars-d-learn
On Wed, Jun 17, 2020 at 11:50:27AM +, Per Nordlöw via Digitalmars-d-learn wrote: > Should a range-compliant aggregate type realizing a parser be encoded > as a struct or class? Preferably a struct IMO, but see below. > In dmd `Lexer` and `Parser` are both classes. Probably for h

Re: Should a parser type be a struct or class?

2020-06-17 Thread Stanislav Blinov via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 11:50:27 UTC, Per Nordlöw wrote: Should a range-compliant aggregate type realizing a parser be encoded as a struct or class? In dmd `Lexer` and `Parser` are both classes. In general how should I reason about whether an aggregate type should be encoded as a

Re: Should a parser type be a struct or class?

2020-06-17 Thread Simen Kjærås via Digitalmars-d-learn
On Wednesday, 17 June 2020 at 11:50:27 UTC, Per Nordlöw wrote: Should a range-compliant aggregate type realizing a parser be encoded as a struct or class? In dmd `Lexer` and `Parser` are both classes. In general how should I reason about whether an aggregate type should be encoded as a

Should a parser type be a struct or class?

2020-06-17 Thread Per Nordlöw via Digitalmars-d-learn
Should a range-compliant aggregate type realizing a parser be encoded as a struct or class? In dmd `Lexer` and `Parser` are both classes. In general how should I reason about whether an aggregate type should be encoded as a struct or class?

Re: Why Dlang use parsing expression grammar (PEG) parser not BNF?

2019-11-01 Thread user4567 via Digitalmars-d-learn
On Thursday, 31 October 2019 at 08:40:42 UTC, lili wrote: Hi: I want implementation Lua on D, I find that a PEG parser https://github.com/PhilippeSigaud/Pegged why do not use BNF parser. Is PEG better than BNF? IIRC the PEG for D is not complete. You have an EBNF here https

Re: Why Dlang use parsing expression grammar (PEG) parser not BNF?

2019-11-01 Thread Dennis via Digitalmars-d-learn
On Thursday, 31 October 2019 at 08:40:42 UTC, lili wrote: Hi: I want implementation Lua on D, I find that a PEG parser https://github.com/PhilippeSigaud/Pegged why do not use BNF parser. Is PEG better than BNF? The readme has a link to the reference article: https://bford.info/pub

Why Dlang use parsing expression grammar (PEG) parser not BNF?

2019-10-31 Thread lili via Digitalmars-d-learn
Hi: I want implementation Lua on D, I find that a PEG parser https://github.com/PhilippeSigaud/Pegged why do not use BNF parser. Is PEG better than BNF?

Re: fasta parser with iopipe?

2017-08-30 Thread Steven Schveighoffer via Digitalmars-d-learn
On 8/28/17 10:08 AM, biocyberman wrote: @Steve: Yes we talked at dconf 2017. I had to other things so D learning got slow down. I am trying with Fasta format before jumping to Fastq again. The jsoniopipe is full feature, and relatively small project, which can be used to study case. However th

Re: fasta parser with iopipe?

2017-08-28 Thread biocyberman via Digitalmars-d-learn
On Wednesday, 23 August 2017 at 13:06:36 UTC, Steven Schveighoffer wrote: On 8/23/17 5:53 AM, biocyberman wrote: [...] I'll respond to all your questions with what I would do, instead of answering each one. I would suggest an approach similar to how I approached parsing JSON data. In your

Re: fasta parser with iopipe?

2017-08-23 Thread Steven Schveighoffer via Digitalmars-d-learn
e can be plain-text or gzip compressed. # Goals: Write a parser that uses Dlang range with iopipe library for performance and ease of use. A big fasta file can be dozens of gigabytes. # Questions: 1. How do I model a fasta entry with a struct or class? 2. How to I implement a range of fasta entries

Re: fasta parser with iopipe?

2017-08-23 Thread Nicholas Wilson via Digitalmars-d-learn
l regex). (if you know your sequence has no Ns or other ambiguous bases you can can store 4 bases to a byte, or 3 if you want them in triplets) - A fasta file can be plain-text or gzip compressed. # Goals: Write a parser that uses Dlang range with iopipe library for performance and ease of

fasta parser with iopipe?

2017-08-23 Thread biocyberman via Digitalmars-d-learn
ible. - Headers may contain annotation information separated by some delimiter (i.e. | in this case). - Entry ID and header is a single line, which does not contain newline characters. - Sequence under the header line is [ATCGN\n]* (Perl regex). - A fasta file can be plain-text or gzip compressed. #

Re: html fetcher/parser

2017-08-14 Thread Adam D. Ruppe via Digitalmars-d-learn
On Monday, 14 August 2017 at 23:15:13 UTC, Faux Amis wrote: (Althought following the spec would be the first step to a D html layout engine :D ) Oh, I've actually done some of that before too. https://github.com/adamdruppe/arsd/blob/master/htmlwidget.d It is pretty horrible... but managed to

Re: html fetcher/parser

2017-08-14 Thread Faux Amis via Digitalmars-d-learn
On 2017-08-13 19:51, Adam D. Ruppe wrote: On Sunday, 13 August 2017 at 15:54:45 UTC, Faux Amis wrote: Just curious, but is there a spec of sorts which defines which errors should be fixed and such? The HTML5 spec describes how you are supposed to parse various things, including the recovery p

Re: html fetcher/parser

2017-08-13 Thread Adam D. Ruppe via Digitalmars-d-learn
On Sunday, 13 August 2017 at 15:54:45 UTC, Faux Amis wrote: Just curious, but is there a spec of sorts which defines which errors should be fixed and such? The HTML5 spec describes how you are supposed to parse various things, including the recovery paths for broken markup. My module, howeve

Re: html fetcher/parser

2017-08-13 Thread Faux Amis via Digitalmars-d-learn
On 2017-08-13 01:49, Soulsbane wrote: On Saturday, 12 August 2017 at 19:53:22 UTC, Faux Amis wrote: I would like to get into D again by making a small program which fetches a website every X-time and keeps track of all changes within specified dom elements. fetching: should I go for std curl,

Re: html fetcher/parser

2017-08-13 Thread Faux Amis via Digitalmars-d-learn
On 2017-08-12 22:22, Adam D. Ruppe wrote: On Saturday, 12 August 2017 at 19:53:22 UTC, Faux Amis wrote: [...] [...] --- // compile: $ dmd thisfile.d ~/arsd/{dom,http2,characterencodings} import std.stdio; import arsd.dom; void main() { auto document = Document.fromUrl("https://dlang

Re: html fetcher/parser

2017-08-12 Thread Soulsbane via Digitalmars-d-learn
On Saturday, 12 August 2017 at 19:53:22 UTC, Faux Amis wrote: I would like to get into D again by making a small program which fetches a website every X-time and keeps track of all changes within specified dom elements. fetching: should I go for std curl, vibe.d or something else? parsing: I c

Re: html fetcher/parser

2017-08-12 Thread Michael via Digitalmars-d-learn
On Saturday, 12 August 2017 at 20:22:44 UTC, Adam D. Ruppe wrote: On Saturday, 12 August 2017 at 19:53:22 UTC, Faux Amis wrote: [...] My dom.d and http2.d combine to make this easy: https://github.com/adamdruppe/arsd/blob/master/dom.d https://github.com/adamdruppe/arsd/blob/master/http2.d [.

Re: html fetcher/parser

2017-08-12 Thread Adam D. Ruppe via Digitalmars-d-learn
On Saturday, 12 August 2017 at 19:53:22 UTC, Faux Amis wrote: I would like to get into D again by making a small program which fetches a website every X-time and keeps track of all changes within specified dom elements. My dom.d and http2.d combine to make this easy: https://github.com/adamdr

html fetcher/parser

2017-08-12 Thread Faux Amis via Digitalmars-d-learn
I would like to get into D again by making a small program which fetches a website every X-time and keeps track of all changes within specified dom elements. fetching: should I go for std curl, vibe.d or something else? parsing: I could only find these dub packages: htmld & libdominator. And th

Re: How to write parser?

2017-05-15 Thread crimaniak via Digitalmars-d-learn
On Sunday, 14 May 2017 at 19:00:09 UTC, Suliman wrote: I am trying to learn how to write text parser. I have example doc with follow format: #Header my header text ##SubHeader my sub header text it's there any better way to parse such format? Ready markdown Pegged based parser:

How to write parser?

2017-05-14 Thread Suliman via Digitalmars-d-learn
I am trying to learn how to write text parser. I have example doc with follow format: #Header my header text ##SubHeader my sub header text ###Sub3Header my sub 3 text #Header21 my header2 text ##SubHeader21 my header2 text ###SubHeader22 my header3 text I would like to wrap all level

Re: D Grammar Parser In D

2016-02-19 Thread cym13 via Digitalmars-d-learn
don't want to parse d code. I want a generic parser that is written in D like how irony is written in C#. Pegged maybe? https://github.com/PhilippeSigaud/Pegged

Re: D Grammar Parser In D

2016-02-19 Thread Patience via Digitalmars-d-learn
On Friday, 19 February 2016 at 22:34:22 UTC, Adam D. Ruppe wrote: On Friday, 19 February 2016 at 22:29:32 UTC, Patience wrote: Is there anything in D like https://irony.codeplex.com/ https://github.com/Hackerpilot/libdparse/ I don't want to parse d code. I want a generic parser th

Re: D Grammar Parser In D

2016-02-19 Thread Adam D. Ruppe via Digitalmars-d-learn
On Friday, 19 February 2016 at 22:29:32 UTC, Patience wrote: Is there anything in D like https://irony.codeplex.com/ https://github.com/Hackerpilot/libdparse/

D Grammar Parser In D

2016-02-19 Thread Patience via Digitalmars-d-learn
Is there anything in D like https://irony.codeplex.com/

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 18:42:29 UTC, Andrew Brown wrote: I had some luck building a local copy of llvm in my home directory, using a linux version about as old as yours (llvm 3.5 i used) specifying: --configure --prefix=/home/andrew/llvm so make install would install it somewhere I

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Andrew Brown via Digitalmars-d-learn
I had some luck building a local copy of llvm in my home directory, using a linux version about as old as yours (llvm 3.5 i used) specifying: --configure --prefix=/home/andrew/llvm so make install would install it somewhere I had permissions. Then I changed the cmake command to: cmake -L -DL

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread H. S. Teoh via Digitalmars-d-learn
On Tue, Sep 15, 2015 at 08:55:43AM +, Fredrik Boulund via Digitalmars-d-learn wrote: > On Monday, 14 September 2015 at 18:31:38 UTC, H. S. Teoh wrote: > >I tried implementing a crude version of this (see code below), and > >found that manually calling GC.collect() even as frequently as once >

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread John Colvin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 13:49:04 UTC, Fredrik Boulund wrote: On Tuesday, 15 September 2015 at 10:01:30 UTC, John Colvin wrote: [...] Nope, :( [...] Oh well, worth a try I guess.

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 10:01:30 UTC, John Colvin wrote: try this: https://dlangscience.github.io/resources/ldc-0.16.0-a2_glibc2.11.3.tar.xz Nope, :( $ ldd ldc2 ./ldc2: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./ldc2) linux-vdso.so.1 => (0x

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread John Colvin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 13:01:06 UTC, Kagamin wrote: On Tuesday, 15 September 2015 at 09:19:29 UTC, John Colvin wrote: It provides you only one char at a time instead of a whole line. It will be quite constraining for your code if not mind-bending. http://dlang.org/phobos/std_string.

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Kagamin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 09:19:29 UTC, John Colvin wrote: It provides you only one char at a time instead of a whole line. It will be quite constraining for your code if not mind-bending. http://dlang.org/phobos/std_string.html#.lineSplitter File(fileName).byChunk(chunkSize).map!"cast

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread John Colvin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 08:45:00 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 15:04:12 UTC, John Colvin wrote: [...] Thanks for the offer, but don't go out of your way for my sake. Maybe I'll just build this in a clean environment instead of on my work computer to get

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread John Colvin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 09:09:00 UTC, Kagamin wrote: On Tuesday, 15 September 2015 at 08:53:37 UTC, Fredrik Boulund wrote: my favourite for streaming a file: enum chunkSize = 4096; File(fileName).byChunk(chunkSize).map!"cast(char[])a".joiner() Is this an efficient way of reading this

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Kagamin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 08:53:37 UTC, Fredrik Boulund wrote: my favourite for streaming a file: enum chunkSize = 4096; File(fileName).byChunk(chunkSize).map!"cast(char[])a".joiner() Is this an efficient way of reading this type of file? What should one keep in mind when choosing chunk

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Rikki Cattermole via Digitalmars-d-learn
On 15/09/15 9:00 PM, Kagamin wrote: On Tuesday, 15 September 2015 at 08:51:02 UTC, Fredrik Boulund wrote: Using char[] all around might be a good idea, but it doesn't seem like the string conversions are really that taxing. What are the arguments for working on char[] arrays rather than strings?

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Kagamin via Digitalmars-d-learn
On Tuesday, 15 September 2015 at 08:51:02 UTC, Fredrik Boulund wrote: Using char[] all around might be a good idea, but it doesn't seem like the string conversions are really that taxing. What are the arguments for working on char[] arrays rather than strings? No, casting to string would be i

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 18:31:38 UTC, H. S. Teoh wrote: I tried implementing a crude version of this (see code below), and found that manually calling GC.collect() even as frequently as once every 5000 loop iterations (for a 500,000 line test input file) still gives about 15% performanc

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 16:33:23 UTC, Rikki Cattermole wrote: A lot of this hasn't been covered I believe. http://dpaste.dzfl.pl/f7ab2915c3e1 1) You don't need to convert char[] to string via to. No. Too much. Cast it. 2) You don't need byKey, use foreach key, value syntax. That way

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 18:08:31 UTC, John Colvin wrote: On Monday, 14 September 2015 at 17:51:43 UTC, CraigDillabaugh wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] I am going to go off the beaten path here. If you really want speed for a file lik

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 16:13:14 UTC, Edwin van Leeuwen wrote: See this link for clarification on what the columns/numbers in the profile file mean http://forum.dlang.org/post/f9gjmo$2gce$1...@digitalmars.com It is still difficult to parse though. I myself often use sysprof (only avail

Re: Speeding up text file parser (BLAST tabular format)

2015-09-15 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 15:04:12 UTC, John Colvin wrote: I've had nothing but trouble when using different versions of libc. It would be easier to do this instead: http://wiki.dlang.org/Building_LDC_from_source I'm running a build of LDC git HEAD right now on an old server with 2.11

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Rikki Cattermole via Digitalmars-d-learn
On 15/09/15 5:41 AM, NX wrote: On Monday, 14 September 2015 at 16:33:23 UTC, Rikki Cattermole wrote: A lot of this hasn't been covered I believe. http://dpaste.dzfl.pl/f7ab2915c3e1 I believe that should be: foreach (query, ref value; hitlists) Since an assignment happenin there..? Probably.

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread H. S. Teoh via Digitalmars-d-learn
On Mon, Sep 14, 2015 at 08:07:45PM +, Kapps via Digitalmars-d-learn wrote: > On Monday, 14 September 2015 at 18:31:38 UTC, H. S. Teoh wrote: > >I decided to give the code a spin with `gdc -O3 -pg`. Turns out that > >the hotspot is in std.array.split, contrary to expectations. :-) > >Here are th

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Kapps via Digitalmars-d-learn
On Monday, 14 September 2015 at 18:31:38 UTC, H. S. Teoh wrote: I decided to give the code a spin with `gdc -O3 -pg`. Turns out that the hotspot is in std.array.split, contrary to expectations. :-) Here are the first few lines of the gprof output: [...] Perhaps using the new rangified spli

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread H. S. Teoh via Digitalmars-d-learn
I decided to give the code a spin with `gdc -O3 -pg`. Turns out that the hotspot is in std.array.split, contrary to expectations. :-) Here are the first few lines of the gprof output: -snip- Each sample counts as 0.01 seconds. % cumulative self self total

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 17:51:43 UTC, CraigDillabaugh wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] I am going to go off the beaten path here. If you really want speed for a file like this one way of getting that is to read the file in as a single

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread CraigDillabaugh via Digitalmars-d-learn
now. I had this idea that it'd be fun to write a parser for a text-based tabular data format I tend to read a lot of in my programs, but I was a bit stomped that the D implementation I created was slower than my Python-version. I tried running `dmd -profile` on it but didn't really under

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread NX via Digitalmars-d-learn
On Monday, 14 September 2015 at 16:33:23 UTC, Rikki Cattermole wrote: A lot of this hasn't been covered I believe. http://dpaste.dzfl.pl/f7ab2915c3e1 I believe that should be: foreach (query, ref value; hitlists) Since an assignment happenin there..?

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 16:33:23 UTC, Rikki Cattermole wrote: On 15/09/15 12:30 AM, Fredrik Boulund wrote: [...] A lot of this hasn't been covered I believe. http://dpaste.dzfl.pl/f7ab2915c3e1 1) You don't need to convert char[] to string via to. No. Too much. Cast it. Not a good

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread H. S. Teoh via Digitalmars-d-learn
On Mon, Sep 14, 2015 at 04:13:12PM +, Edwin van Leeuwen via Digitalmars-d-learn wrote: > On Monday, 14 September 2015 at 14:54:34 UTC, Fredrik Boulund wrote: > >[...] I tried using the built-in profiler in DMD on the D program but > >to no avail. I couldn't really make any sense of the output

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Rikki Cattermole via Digitalmars-d-learn
x27;d be fun to write a parser for a text-based tabular data format I tend to read a lot of in my programs, but I was a bit stomped that the D implementation I created was slower than my Python-version. I tried running `dmd -profile` on it but didn't really understand what I can do to make i

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Edwin van Leeuwen via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:54:34 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 14:40:29 UTC, H. S. Teoh wrote: I agree with you on that. I used Python's cProfile module to find the performance bottleneck in the Python version I posted, and shaved off 8-10 seconds of runtim

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:35:26 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 14:28:41 UTC, John Colvin wrote: Yup, glibc is too old for those binaries. What does "ldd --version" say? It says "ldd (GNU libc) 2.12". Hmm... The most recent version in RHEL's repo is "2.12

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:40:29 UTC, H. S. Teoh wrote: If performance is a problem, the first thing I'd recommend is to use a profiler to find out where the hotspots are. (More often than not, I have found that the hotspots are not where I expected them to be; sometimes a 1-line change

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread H. S. Teoh via Digitalmars-d-learn
On Mon, Sep 14, 2015 at 02:34:41PM +, Fredrik Boulund via Digitalmars-d-learn wrote: > On Monday, 14 September 2015 at 14:18:58 UTC, John Colvin wrote: > >Range-based code like you are using leads to *huge* numbers of > >function calls to get anything done. The advantage of inlining is > >twof

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:28:41 UTC, John Colvin wrote: Yup, glibc is too old for those binaries. What does "ldd --version" say? It says "ldd (GNU libc) 2.12". Hmm... The most recent version in RHEL's repo is "2.12-1.166.el6_7.1", which is what is installed. Can this be side-loaded

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:18:58 UTC, John Colvin wrote: Range-based code like you are using leads to *huge* numbers of function calls to get anything done. The advantage of inlining is twofold: 1) you don't have to pay the cost of the function call itself and 2) often more optimisation

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:15:25 UTC, Laeeth Isharc wrote: I picked up D to start learning maybe a couple of years ago. I found Ali's book, Andrei's book, github source code (including for Phobos), and asking here to be the best resources. The docs make perfect sense when you have got

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:25:04 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 14:14:18 UTC, John Colvin wrote: what system are you on? What are the error messages you are getting? I really appreciate your will to try to help me out. This is what ldd shows on the latest

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 14:14:18 UTC, John Colvin wrote: what system are you on? What are the error messages you are getting? I really appreciate your will to try to help me out. This is what ldd shows on the latest binary release of LDC on my machine. I'm on a Red Hat Enterprise Linu

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:58:33 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 13:37:18 UTC, John Colvin wrote: On Monday, 14 September 2015 at 13:05:32 UTC, Andrea Fontana wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] Also if problem

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Laeeth Isharc via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:55:50 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 13:10:50 UTC, Edwin van Leeuwen wrote: Two things that you could try: First hitlists.byKey can be expensive (especially if hitlists is big). Instead use: foreach( key, value ; hitlists ) Als

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:50:22 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 13:05:32 UTC, Andrea Fontana wrote: [...] Thanks for the suggestions! I'm not too familiar with compiled languages like this, I've mainly written small programs in D and run them via `rdmd` i

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:10:50 UTC, Edwin van Leeuwen wrote: Two things that you could try: First hitlists.byKey can be expensive (especially if hitlists is big). Instead use: foreach( key, value ; hitlists ) Also the filter.array.length is quite expensive. You could use count ins

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:37:18 UTC, John Colvin wrote: On Monday, 14 September 2015 at 13:05:32 UTC, Andrea Fontana wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] Also if problem probabily is i/o related, have you tried with: -O -inline -release -n

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Fredrik Boulund via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:05:32 UTC, Andrea Fontana wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] Also if problem probabily is i/o related, have you tried with: -O -inline -release -noboundscheck ? Anyway I think it's a good idea to test it against

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread John Colvin via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:05:32 UTC, Andrea Fontana wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] Also if problem probabily is i/o related, have you tried with: -O -inline -release -noboundscheck ? -inline in particular is likely to have a strong

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Edwin van Leeuwen via Digitalmars-d-learn
On Monday, 14 September 2015 at 12:50:03 UTC, Fredrik Boulund wrote: On Monday, 14 September 2015 at 12:44:22 UTC, Edwin van Leeuwen wrote: Sounds like this program is actually IO bound. In that case I would not expect a really expect an improvement by using D. What is the CPU usage like when y

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Andrea Fontana via Digitalmars-d-learn
On Monday, 14 September 2015 at 13:05:32 UTC, Andrea Fontana wrote: On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] Also if problem probabily is i/o related, have you tried with: -O -inline -release -noboundscheck ? Anyway I think it's a good idea to test it against

Re: Speeding up text file parser (BLAST tabular format)

2015-09-14 Thread Andrea Fontana via Digitalmars-d-learn
On Monday, 14 September 2015 at 12:30:21 UTC, Fredrik Boulund wrote: [...] Also if problem probabily is i/o related, have you tried with: -O -inline -release -noboundscheck ? Anyway I think it's a good idea to test it against gdc and ldc that are known to generate faster executables. Andrea

  1   2   >