[
https://issues.apache.org/jira/browse/ARROW-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245180#comment-16245180
]
ASF GitHub Bot commented on ARROW-1693:
---------------------------------------
trxcllnt opened a new pull request #1294: WIP ARROW-1693: Fix reading C++
dictionary-encoded vectors
URL: https://github.com/apache/arrow/pull/1294
This PR adds a workaround for reading the metadata layout for C++
dictionary-encoded vectors.
I added tests that validate against the C++/Java integration suite. In order
to make the new tests pass, I had to update the generated flatbuffers format
and add a few types the JS version didn't have yet (Bool, Date32, and
Timestamp). It also uses the new `isDelta` flag on DictionaryBatches to
determine whether the DictionaryBatch vector should replace or append to the
existing dictionary.
I also added a script for generating test arrow files from the C++ and Java
implementations, so we don't break the tests updating the format in the future.
I saved the generated Arrow files in with the tests because I didn't see a way
to pipe the JSON test data through the C++/Java json-to-arrow commands without
writing to a file. If I missed something and we can do it all in-memory, I'd be
happy to make that change!
This PR is marked WIP because I added an [integration
test](https://github.com/apache/arrow/commit/6e98874d9f4bfae7758f8f731212ae7ceb3f1321#diff-18c6be12406c482092d4b1f7bd70a8e1R22)
that validates the JS reader reads C++ and Java files the same way, but
unfortunately it doesn't. Debugging, I noticed a number of other differences
between the buffer layout metadata between the C++ and Java versions. If we go
ahead with @jacques-n [comment in
ARROW-1693](https://issues.apache.org/jira/browse/ARROW-1693?focusedCommentId=16244812&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16244812)
and remove/ignore the metadata, this test should pass too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [JS] Error reading dictionary-encoded integration test files
> ------------------------------------------------------------
>
> Key: ARROW-1693
> URL: https://issues.apache.org/jira/browse/ARROW-1693
> Project: Apache Arrow
> Issue Type: Bug
> Components: JavaScript
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Labels: pull-request-available
> Fix For: 0.8.0
>
> Attachments: dictionary-cpp.arrow, dictionary-java.arrow,
> dictionary.json
>
>
> The JS implementation crashes when reading the dictionary test case from the
> integration tests.
> To replicate, first generate the test files with java and cpp impls:
> {code}
> $ cd ${ARROW_HOME}/integration/
> $ python -c 'from integration_test import generate_dictionary_case;
> generate_dictionary_case().write("dictionary.json")'
> $ ../cpp/debug/debug/json-integration-test --integration
> --json=dictionary.json --arrow=dictionary-cpp.arrow --mode=JSON_TO_ARROW
> $ java -cp
> ../java/tools/target/arrow-tools-0.8.0-SNAPSHOT-jar-with-dependencies.jar
> org.apache.arrow.tools.Integration -c JSON_TO_ARROW -a dictionary-java.arrow
> -j dictionary.json
> {code}
> Attempt to read the files with the JS impl:
> {code}
> $ cd ${ARROW_HOME}/js/
> $ ./bin/arrow2csv.js -s dict1_0 -f ../integration/dictionary-{java,cpp}.arrow
> {code}
> Both files result in an error for me on
> [a8f51858|https://github.com/apache/arrow/commit/a8f518588fda471b2e3cc8e0f0064e7c4bb99899]:
> {{TypeError: Cannot read property 'buffer' of undefined}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
