Parquet files with microsecond columns

2024-01-22 Thread Peter Franzen
Hi, I am using Drill to query Parquet files that have fields of type timestamp_micros. By default, Drill truncates those microsecond values to milliseconds when reading the Parquet files in order to convert them to SQL timestamps. In some of my use cases I need to read the original microsecond

DFDL Standard approved for ISO JTC1 PAS (Pre-Approved Standard)

2024-01-22 Thread Mike Beckerle
I received notice today that the DFDL OGF standard is officially headed to become an ISO standard. The ballot within ISO JTC1 passed 100%. I will let you know more info when I get it about how we can propagate this information, any branding considerations ISO JTC1 requires about it, their trademar

Re: DFDL Standard approved for ISO JTC1 PAS (Pre-Approved Standard)

2024-01-22 Thread Charles Givre
Nice! > On Jan 22, 2024, at 08:44, Mike Beckerle wrote: > > I received notice today that the DFDL OGF standard is officially headed to > become an ISO standard. The ballot within ISO JTC1 passed 100%. > > I will let you know more info when I get it about how we can propagate this > information,

Re: Parquet files with microsecond columns

2024-01-22 Thread Paul Rogers
Hi Peter, It sounds like you are on the right track: the new option is the quick short-term solution. The best long-term solution is to generalize Drill's date/time type, but that would take much more work. (Drill also has a bug where the treatment of timezones is incorrect, which forces Drill to

[I] union duplicate key (drill)

2024-01-22 Thread via GitHub
yaozhu opened a new issue, #2873: URL: https://github.com/apache/drill/issues/2873 mysql table baseall and table test has the same tale struct and recored show create table baseall; +-+-

Re: [I] union duplicate key (drill)

2024-01-22 Thread via GitHub
cgivre commented on issue #2873: URL: https://github.com/apache/drill/issues/2873#issuecomment-1905101608 Can you please provide a screenshot or some explanation that is more clear? This is really difficult to follow. -- This is an automated message from the Apache Git Service. To respon

Re: [I] union duplicate key (drill)

2024-01-22 Thread via GitHub
yaozhu commented on issue #2873: URL: https://github.com/apache/drill/issues/2873#issuecomment-1905105700 > Can you please provide a screenshot or some explanation that is more clear? This is really difficult to follow. ![image](https://github.com/apache/drill/assets/13361571/82b08db

Re: [I] union duplicate key (drill)

2024-01-22 Thread via GitHub
yaozhu commented on issue #2873: URL: https://github.com/apache/drill/issues/2873#issuecomment-1905126799 @cgivre are u clear? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] union duplicate key (drill)

2024-01-22 Thread via GitHub
cgivre commented on issue #2873: URL: https://github.com/apache/drill/issues/2873#issuecomment-1905132740 I'm not yet convinced there is a bug in Drill here. I think the issue may actually be in your query. My hunch here is that there may be an issue in the `ORDER BY` statement. Your inn

Re: [I] union duplicate key (drill)

2024-01-22 Thread via GitHub
yaozhu commented on issue #2873: URL: https://github.com/apache/drill/issues/2873#issuecomment-1905216330 > I'm not yet convinced there is a bug in Drill here. I think the issue may actually be in your query. My hunch here is that there may be an issue in the `ORDER BY` statement. Your inne

Re: [I] union duplicate key (drill)

2024-01-22 Thread via GitHub
yaozhu commented on issue #2873: URL: https://github.com/apache/drill/issues/2873#issuecomment-1905217272 This is a test case written by testers, without actual usage scenarios. What do you mean by combining them? -- This is an automated message from the Apache Git Service. To respond

[I] hashpartition leak (drill)

2024-01-22 Thread via GitHub
shfshihuafeng opened a new issue, #2874: URL: https://github.com/apache/drill/issues/2874 Before submitting a bug report, please verify that you are using the most current version of Drill. **Describe the bug** hashpartition leak when allocate memory exception with OutOfMemoryEx

Re: [I] hashpartition leak (drill)

2024-01-22 Thread via GitHub
shfshihuafeng commented on issue #2874: URL: https://github.com/apache/drill/issues/2874#issuecomment-1905298953 I fixed the bug see attachment [0001-bug-10227615-hashPartition-leak.patch](https://github.com/apache/drill/files/14018533/0001-bug-10227615-hashPartition-leak.patch)

Re: [I] hashpartition leak (drill)

2024-01-22 Thread via GitHub
cgivre commented on issue #2874: URL: https://github.com/apache/drill/issues/2874#issuecomment-1905299722 @shfshihuafeng Can you please submit this as a pull request? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] the mergejoin memery leak (drill)

2024-01-22 Thread via GitHub
shfshihuafeng commented on issue #2871: URL: https://github.com/apache/drill/issues/2871#issuecomment-1905300921 > @weijunlu, you may have found a bug. The behavior you described is not expected. > > Just to verify I understand: you ran your query once. Did the query run successfully

Re: [I] hashpartition leak (drill)

2024-01-22 Thread via GitHub
shfshihuafeng commented on issue #2874: URL: https://github.com/apache/drill/issues/2874#issuecomment-1905301573 ok @cgivre -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[jira] [Created] (DRILL-8478) mergejion memory leak when exception

2024-01-22 Thread shihuafeng (Jira)
shihuafeng created DRILL-8478: - Summary: mergejion memory leak when exception Key: DRILL-8478 URL: https://issues.apache.org/jira/browse/DRILL-8478 Project: Apache Drill Issue Type: Bug

[PR] DRILL-8478. HashPartition memory leak when it allocate memory exception with OutOfMemoryException (#2874) (drill)

2024-01-22 Thread via GitHub
shfshihuafeng opened a new pull request, #2875: URL: https://github.com/apache/drill/pull/2875 # [DRILL-](https://issues.apache.org/jira/browse/DRILL-): PR Title DRILL-8478. HashPartition memory leak when it allocate memory exception with OutOfMemoryException (#2874) ##

Re: [PR] DRILL-8478. HashPartition memory leak when it allocate memory exception with OutOfMemoryException (#2874) (drill)

2024-01-22 Thread via GitHub
paul-rogers commented on code in PR #2875: URL: https://github.com/apache/drill/pull/2875#discussion_r1462817821 ## exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java: ## @@ -157,11 +162,11 @@ public HashPartition(FragmentContext context,

Re: [PR] DRILL-8478. HashPartition memory leak when it allocate memory exception with OutOfMemoryException (#2874) (drill)

2024-01-22 Thread via GitHub
paul-rogers commented on code in PR #2875: URL: https://github.com/apache/drill/pull/2875#discussion_r1462817821 ## exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java: ## @@ -157,11 +162,11 @@ public HashPartition(FragmentContext context,

Re: [PR] DRILL-8478. HashPartition memory leak when it allocate memory exception with OutOfMemoryException (#2874) (drill)

2024-01-22 Thread via GitHub
shfshihuafeng commented on code in PR #2875: URL: https://github.com/apache/drill/pull/2875#discussion_r1462854154 ## exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java: ## @@ -157,11 +162,11 @@ public HashPartition(FragmentContext context