[jira] [Updated] (ARROW-606) [C++] Upgrade to flatbuffers 1.6.0

2017-03-09 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-606: --- Issue Type: New Feature (was: Bug) > [C++] Upgrade to flatbuffers 1.6.0 >

[jira] [Created] (ARROW-606) [C++] Upgrade to flatbuffers 1.6.0

2017-03-09 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-606: -- Summary: [C++] Upgrade to flatbuffers 1.6.0 Key: ARROW-606 URL: https://issues.apache.org/jira/browse/ARROW-606 Project: Apache Arrow Issue Type: Bug C

[jira] [Created] (ARROW-607) [C++] Speed up bitsetting in ArrayBuilder::UnsafeSetNotNull

2017-03-09 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-607: -- Summary: [C++] Speed up bitsetting in ArrayBuilder::UnsafeSetNotNull Key: ARROW-607 URL: https://issues.apache.org/jira/browse/ARROW-607 Project: Apache Arrow I

[jira] [Created] (ARROW-608) [Format] Days since epoch date type

2017-03-09 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-608: -- Summary: [Format] Days since epoch date type Key: ARROW-608 URL: https://issues.apache.org/jira/browse/ARROW-608 Project: Apache Arrow Issue Type: New Feature

[jira] [Created] (ARROW-609) [C++] Function for casting from days since UNIX epoch to int64 date

2017-03-09 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-609: -- Summary: [C++] Function for casting from days since UNIX epoch to int64 date Key: ARROW-609 URL: https://issues.apache.org/jira/browse/ARROW-609 Project: Apache Arrow

[jira] [Created] (ARROW-610) [C++] Win32 compatibility in file.cc

2017-03-09 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-610: -- Summary: [C++] Win32 compatibility in file.cc Key: ARROW-610 URL: https://issues.apache.org/jira/browse/ARROW-610 Project: Apache Arrow Issue Type: New Feature

[jira] [Commented] (ARROW-488) [Python] Implement conversion between integer coded as floating points with NaN to an Arrow integer type

2017-03-09 Thread Miki Tebeka (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903199#comment-15903199 ] Miki Tebeka commented on ARROW-488: --- Is the dtype still integer? I see that Pandas change

[jira] [Commented] (ARROW-488) [Python] Implement conversion between integer coded as floating points with NaN to an Arrow integer type

2017-03-09 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903242#comment-15903242 ] Wes McKinney commented on ARROW-488: [~tebeka] the pandas behavior is the motivation fo

[jira] [Commented] (ARROW-413) DATE type is not specified clearly

2017-03-09 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903535#comment-15903535 ] Bryan Cutler commented on ARROW-413: I started working on ARROW-582, to add Date/Time t

[jira] [Commented] (ARROW-413) DATE type is not specified clearly

2017-03-09 Thread Uwe L. Korn (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903544#comment-15903544 ] Uwe L. Korn commented on ARROW-413: --- [~bryanc] yes, DATE is milliseconds since 1970 but o

Date/Time fields values in Java

2017-03-09 Thread Bryan Cutler
Hello All, I've started work on ARROW-582 to add Date/Time support for Java JSON files and would just like to clear up a few things. I believe the Java Time type is supposed to represent milliseconds since epoch, it is stored as a FixedValueVector with a width of 4 bytes (equivalent to Java 'int'

[jira] [Assigned] (ARROW-605) [C++] Refactor generic ArrayLoader class, support work for Feather merge

2017-03-09 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reassigned ARROW-605: -- Assignee: Wes McKinney > [C++] Refactor generic ArrayLoader class, support work for Feather merg

[jira] [Resolved] (ARROW-605) [C++] Refactor generic ArrayLoader class, support work for Feather merge

2017-03-09 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-605. Resolution: Fixed Issue resolved by pull request 365 [https://github.com/apache/arrow/pull/365] > [C

[jira] [Updated] (ARROW-605) [C++] Refactor generic ArrayLoader class, support work for Feather merge

2017-03-09 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-605: --- Fix Version/s: 0.3.0 > [C++] Refactor generic ArrayLoader class, support work for Feather merge > -

[jira] [Commented] (ARROW-607) [C++] Speed up bitsetting in ArrayBuilder::UnsafeSetNotNull

2017-03-09 Thread Robert Nishihara (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903668#comment-15903668 ] Robert Nishihara commented on ARROW-607: I believe this is already done in ARROW-55

Re: Date/Time fields values in Java

2017-03-09 Thread Bryan Cutler
I guess it would make sense to just store the time of day value in milliseconds to go along with the DATE type that contains days since epoch, which would fit into a 4 byte value. Only I see conflicting code in TypeLayout.java that defines the schema as 64 bit width public TypeLayout visit(Time t

Re: Arrow sync in 10min

2017-03-09 Thread Julien Le Dem
Emilio, Let me know if you need help. On Thu, Mar 2, 2017 at 12:13 PM, Emilio Lahr-Vivaz wrote: > Oops, sorry I mean to attend but forgot. Re: dictionary encoding support, > I haven't had time to work on it recently, but I should have some next > week. I've partially coded some of the changes bu

Re: Date/Time fields values in Java

2017-03-09 Thread Julien Le Dem
Hi Bryan, In the JSON representation we should use the integer representation of the Timestamp. We should not depend on joda for this. DATE is on 8 bytes => 64bits: https://github.com/apache/arrow/blob/6b3ae2aecc8cd31425035a021fa04b9ed3385a8d/format/Message.fbs#L79 https://github.com/apache/arrow/