To clarify, I was suggesting that the order in which the files are read
could be causing the variation in result, not that this is expected. There
definitely seems to be a bug. But the fact that it passes sometimes and not
others suggests the problem is exposed by file ordering.
On Tue, Dec 15, 2015 at 3:31 PM, Amit Hadke <amit.ha...@gmail.com> wrote:
> Jason,
>
> I misunderstood earlier why unit test is failing. It has nothing to do with
> ordering of files.
>
> Whats happening is I'm doing topn operation on field which is union of
> strings and nulls in descending order.
> Test checks if string values are on top but somehow for some people nulls
> are on top and test fails.
>
> I'm suspecting it has to do with how comparator treats null - high/low.
>
> ~ Amit.
>
>
>
> On Tue, Dec 15, 2015 at 3:23 PM, Jason Altekruse <altekruseja...@gmail.com
> >
> wrote:
>
> > Amit,
> >
> > The message out of the test framework tries to provide enough information
> > to debug even if the issues isn't reproducible in your environment. Can
> you
> > think of any reason why it might be giving the different results shown in
> > the message if the order of the batches changed?
> >
> > If you need to change the order yourself there are two hacky approaches
> you
> > could do. Try changing the names or saving the files in a different order
> > to make the FS give them back to you in a different order. You also
> could
> > just combine together the files and adjust the batch cutoff number used
> in
> > the json reader, with various ordering of the records in different
> versions
> > of the dataset.
> >
> > As I write this I realize that combining the files will change the
> behavior
> > of the read. with the first batch giving a single type and later ones
> > giving a union type. As opposed to the multiple files approach which
> would
> > produce a bunch of different individual types and make the sort operation
> > generate the union type. To test this properly we may just need a test
> > harness to produce batches explicitly and feed them into an operator,
> > rather than relying on the JSON reader.
> >
> > - Jason
> >
> > On Tue, Dec 15, 2015 at 2:31 PM, Amit Hadke <amit.ha...@gmail.com>
> wrote:
> >
> > > Hey Guys,
> > >
> > > I'm not able to reproduce same issue and test doesn't seem to be doing
> > > anything.
> > >
> > > Can someone run "mvn -Dtest=TestTopNSchemaChanges#testMissingColumn
> test"
> > > and see if it fails?
> > >
> > > On Mon, Dec 14, 2015 at 11:51 PM, Amit Hadke <amit.ha...@gmail.com>
> > wrote:
> > >
> > > > This seems like a bug in topn code than test.
> > > > We are expecting sorted by kl2 (descending) so that non null values
> > come
> > > > up on top.
> > > > Results seems to be have nulls on top.
> > > >
> > > > ~ Amit.
> > > >
> > > > On Mon, Dec 14, 2015 at 11:27 PM, Jason Altekruse <
> > > > altekruseja...@gmail.com> wrote:
> > > >
> > > >> Seems weird that the results would be different based on reading
> > order,
> > > as
> > > >> the queries themselves contain an order by. Do we return different
> > types
> > > >> out of the sort depending on which schema we get first? Is this
> > > >> intentional?
> > > >>
> > > >> - Jason
> > > >>
> > > >> On Mon, Dec 14, 2015 at 6:06 PM, Steven Phillips <ste...@dremio.com
> >
> > > >> wrote:
> > > >>
> > > >> > I just did a build a linux box, and didn't see this failure. My
> > guess
> > > is
> > > >> > that it fails depending on which order the files are read.
> > > >> >
> > > >> > On Mon, Dec 14, 2015 at 5:38 PM, Venki Korukanti <
> > > >> > venki.koruka...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Is anyone else seeing below failure on latest master? I am
> running
> > > it
> > > >> on
> > > >> > > Linux.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> testMissingColumn(org.apache.drill.exec.physical.impl.TopN.TestTopNSchemaChanges)
> > > >> > > Time elapsed: 2.537 sec <<< ERROR!
> > > >> > > java.lang.Exception: unexpected null at position 0 column
> '`vl2`'
> > > >> should
> > > >> > > have been: 299
> > > >> > >
> > > >> > > Expected Records near verification failure:
> > > >> > > Record Number: 0 { `kl1` : null,`kl2` : 299,`vl2` : 299,`vl1` :
> > > >> > null,`vl` :
> > > >> > > null,`kl` : null, }
> > > >> > > Record Number: 1 { `kl1` : null,`kl2` : 298,`vl2` : 298,`vl1` :
> > > >> > null,`vl` :
> > > >> > > null,`kl` : null, }
> > > >> > > Record Number: 2 { `kl1` : null,`kl2` : 297,`vl2` : 297,`vl1` :
> > > >> > null,`vl` :
> > > >> > > null,`kl` : null, }
> > > >> > >
> > > >> > >
> > > >> > > Actual Records near verification failure:
> > > >> > > Record Number: 0 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1`
> :
> > > >> > null,`vl`
> > > >> > > : 100.0,`kl` : 100.0, }
> > > >> > > Record Number: 1 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1`
> :
> > > >> > null,`vl`
> > > >> > > : 101.0,`kl` : 101.0, }
> > > >> > > Record Number: 2 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1`
> :
> > > >> > null,`vl`
> > > >> > > : 102.0,`kl` : 102.0, }
> > > >> > >
> > > >> > > For query: select kl, vl, kl1, vl1, kl2, vl2 from
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> dfs_test.`/root/drill/exec/java-exec/target/1450142361702-0/topn-schemachanges`
> > > >> > > order by kl2 desc limit 3
> > > >> > > at
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.drill.DrillTestWrapper.compareValuesErrorOnMismatch(DrillTestWrapper.java:512)
> > > >> > > at
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.drill.DrillTestWrapper.compareMergedVectors(DrillTestWrapper.java:170)
> > > >> > > at
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.drill.DrillTestWrapper.compareMergedOnHeapVectors(DrillTestWrapper.java:397)
> > > >> > > at
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.drill.DrillTestWrapper.compareOrderedResults(DrillTestWrapper.java:352)
> > > >> > > at
> > > >> > org.apache.drill.DrillTestWrapper.run(DrillTestWrapper.java:124)
> > > >> > > at org.apache.drill.TestBuilder.go(TestBuilder.java:129)
> > > >> > > at
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.drill.exec.physical.impl.TopN.TestTopNSchemaChanges.testMissingColumn(TestTopNSchemaChanges.java:206)
> > > >> > >
> > > >> > >
> > > >> > > Results :
> > > >> > >
> > > >> > > Tests in error:
> > > >> > > TestTopNSchemaChanges.testMissingColumn:206 ยป unexpected null
> > at
> > > >> > > position 0 c...
> > > >> > >
> > > >> > > Tests run: 4, Failures: 0, Errors: 1, Skipped: 0
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
