Github user lokkju commented on the issue:
https://github.com/apache/spark/pull/16479
So it essentially compiles each implementation against different spark
versions, then *both* bytecodes are included in the final jar? Then reflection
to instantiate it.
That works, without
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16479
Here is a better solution I found:
https://github.com/databricks/spark-avro/pull/217/files#diff-3086eddba29f4034c324541695a2357b
implementing different `OutputWriterFactory` and switch
Github user lokkju commented on the issue:
https://github.com/apache/spark/pull/16479
I'd be interested in the "dirty reflection workarounds", if you have
examples. Not sure how I'd use reflection to handle conflicting interface
definitions, but I'd love to how to.
---
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16479
This is a common issue of the data source v1, it's not powerful enough and
you have to use some Spark internal APIs and hit compatibility problem... AFAIK
a workable solution is to create
Github user lokkju commented on the issue:
https://github.com/apache/spark/pull/16479
So it turns out just copying the conversion code doesn't work, as seen in
spark-avro/#240 - and now I'm running into the same thing writing my own
datasource. As an datasource in the end requires
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/16479
i will just copy the conversion code over for now thx
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16479
Everything in package `org.apache.spark.sql.execution` should be internal
to Spark SQL. Technically you can still implement `OutputWriter` outside of
Spark, but there is no guarantee about the
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/16479
how "internal" are these interfaces really? every time a change like this
is made spark-avro breaks
---
If your project is set up for it, you can reply to this email and have your
reply
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16479
thanks for the review, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16479
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70973/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70973 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70973/testReport)**
for PR 16479 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70973 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70973/testReport)**
for PR 16479 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70971/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70971 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70971/testReport)**
for PR 16479 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70971 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70971/testReport)**
for PR 16479 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70947/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70947 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70947/testReport)**
for PR 16479 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16479
@yhuai It removes unnecessary code to make the codebase easier to maintain.
Besides, the libsvm relation should be a little faster as it doesn't need to go
through a converter.
---
If your
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70947 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70947/testReport)**
for PR 16479 at commit
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/16479
What is the benefit of making these changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16479
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70932/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70932 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70932/testReport)**
for PR 16479 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16479
cc @liancheng @gatorsmile @yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16479
**[Test build #70932 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70932/testReport)**
for PR 16479 at commit
29 matches
Mail list logo
