Amit,
The message out of the test framework tries to provide enough information
to debug even if the issues isn't reproducible in your environment. Can you
think of any reason why it might be giving the different results shown in
the message if the order of the batches changed?
If you need to change the order yourself there are two hacky approaches you
could do. Try changing the names or saving the files in a different order
to make the FS give them back to you in a different order. You also could
just combine together the files and adjust the batch cutoff number used in
the json reader, with various ordering of the records in different versions
of the dataset.
As I write this I realize that combining the files will change the behavior
of the read. with the first batch giving a single type and later ones
giving a union type. As opposed to the multiple files approach which would
produce a bunch of different individual types and make the sort operation
generate the union type. To test this properly we may just need a test
harness to produce batches explicitly and feed them into an operator,
rather than relying on the JSON reader.
- Jason
On Tue, Dec 15, 2015 at 2:31 PM, Amit Hadke <amit.ha...@gmail.com> wrote:
> Hey Guys,
>
> I'm not able to reproduce same issue and test doesn't seem to be doing
> anything.
>
> Can someone run "mvn -Dtest=TestTopNSchemaChanges#testMissingColumn test"
> and see if it fails?
>
> On Mon, Dec 14, 2015 at 11:51 PM, Amit Hadke <amit.ha...@gmail.com> wrote:
>
> > This seems like a bug in topn code than test.
> > We are expecting sorted by kl2 (descending) so that non null values come
> > up on top.
> > Results seems to be have nulls on top.
> >
> > ~ Amit.
> >
> > On Mon, Dec 14, 2015 at 11:27 PM, Jason Altekruse <
> > altekruseja...@gmail.com> wrote:
> >
> >> Seems weird that the results would be different based on reading order,
> as
> >> the queries themselves contain an order by. Do we return different types
> >> out of the sort depending on which schema we get first? Is this
> >> intentional?
> >>
> >> - Jason
> >>
> >> On Mon, Dec 14, 2015 at 6:06 PM, Steven Phillips <ste...@dremio.com>
> >> wrote:
> >>
> >> > I just did a build a linux box, and didn't see this failure. My guess
> is
> >> > that it fails depending on which order the files are read.
> >> >
> >> > On Mon, Dec 14, 2015 at 5:38 PM, Venki Korukanti <
> >> > venki.koruka...@gmail.com>
> >> > wrote:
> >> >
> >> > > Is anyone else seeing below failure on latest master? I am running
> it
> >> on
> >> > > Linux.
> >> > >
> >> > >
> >> > >
> >> >
> >>
> testMissingColumn(org.apache.drill.exec.physical.impl.TopN.TestTopNSchemaChanges)
> >> > > Time elapsed: 2.537 sec <<< ERROR!
> >> > > java.lang.Exception: unexpected null at position 0 column '`vl2`'
> >> should
> >> > > have been: 299
> >> > >
> >> > > Expected Records near verification failure:
> >> > > Record Number: 0 { `kl1` : null,`kl2` : 299,`vl2` : 299,`vl1` :
> >> > null,`vl` :
> >> > > null,`kl` : null, }
> >> > > Record Number: 1 { `kl1` : null,`kl2` : 298,`vl2` : 298,`vl1` :
> >> > null,`vl` :
> >> > > null,`kl` : null, }
> >> > > Record Number: 2 { `kl1` : null,`kl2` : 297,`vl2` : 297,`vl1` :
> >> > null,`vl` :
> >> > > null,`kl` : null, }
> >> > >
> >> > >
> >> > > Actual Records near verification failure:
> >> > > Record Number: 0 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1` :
> >> > null,`vl`
> >> > > : 100.0,`kl` : 100.0, }
> >> > > Record Number: 1 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1` :
> >> > null,`vl`
> >> > > : 101.0,`kl` : 101.0, }
> >> > > Record Number: 2 { `kl1` : null,`vl2` : null,`kl2` : null,`vl1` :
> >> > null,`vl`
> >> > > : 102.0,`kl` : 102.0, }
> >> > >
> >> > > For query: select kl, vl, kl1, vl1, kl2, vl2 from
> >> > >
> >> > >
> >> >
> >>
> dfs_test.`/root/drill/exec/java-exec/target/1450142361702-0/topn-schemachanges`
> >> > > order by kl2 desc limit 3
> >> > > at
> >> > >
> >> > >
> >> >
> >>
> org.apache.drill.DrillTestWrapper.compareValuesErrorOnMismatch(DrillTestWrapper.java:512)
> >> > > at
> >> > >
> >> > >
> >> >
> >>
> org.apache.drill.DrillTestWrapper.compareMergedVectors(DrillTestWrapper.java:170)
> >> > > at
> >> > >
> >> > >
> >> >
> >>
> org.apache.drill.DrillTestWrapper.compareMergedOnHeapVectors(DrillTestWrapper.java:397)
> >> > > at
> >> > >
> >> > >
> >> >
> >>
> org.apache.drill.DrillTestWrapper.compareOrderedResults(DrillTestWrapper.java:352)
> >> > > at
> >> > org.apache.drill.DrillTestWrapper.run(DrillTestWrapper.java:124)
> >> > > at org.apache.drill.TestBuilder.go(TestBuilder.java:129)
> >> > > at
> >> > >
> >> > >
> >> >
> >>
> org.apache.drill.exec.physical.impl.TopN.TestTopNSchemaChanges.testMissingColumn(TestTopNSchemaChanges.java:206)
> >> > >
> >> > >
> >> > > Results :
> >> > >
> >> > > Tests in error:
> >> > > TestTopNSchemaChanges.testMissingColumn:206 ยป unexpected null at
> >> > > position 0 c...
> >> > >
> >> > > Tests run: 4, Failures: 0, Errors: 1, Skipped: 0
> >> > >
> >> >
> >>
> >
> >
>
