bq. have tried these settings with the hbase protocol jar, to no avail
In that case, HBaseZeroCopyByteString is contained in hbase-protocol.jar.
In HBaseZeroCopyByteString , you can see:
package com.google.protobuf; // This is a lie.
If protobuf jar is loaded ahead of hbase-protocol.jar,
I believe I've had trouble with --conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true before, so these might not
work...
I was thinking of trying to add the solr4j jar to
spark.executor.extraClassPath...
On Wed, Sep 30, 2015 at 12:01 PM, Ted Yu
Release of Spark: 1.5.0.
Command line invokation:
ACME_INGEST_HOME=/mnt/acme/acme-ingest
ACME_INGEST_VERSION=0.0.1-SNAPSHOT
ACME_BATCH_DURATION_MILLIS=5000
SPARK_MASTER_URL=spark://data1:7077
JAVA_OPTIONS="-Dspark.streaming.kafka.maxRatePerPartition=1000"
JAVA_OPTIONS="$JAVA_OPTIONS
Mind providing a bit more information:
release of Spark
command line for running Spark job
Cheers
On Tue, Sep 29, 2015 at 1:37 PM, Dmitry Goldenberg wrote:
> We're seeing this occasionally. Granted, this was caused by a wrinkle in
> the Solr schema but this bubbled
Have you tried the following ?
--conf spark.driver.userClassPathFirst=true --conf spark.executor.
userClassPathFirst=true
On Tue, Sep 29, 2015 at 4:38 PM, Dmitry Goldenberg wrote:
> Release of Spark: 1.5.0.
>
> Command line invokation:
>
>
We're seeing this occasionally. Granted, this was caused by a wrinkle in
the Solr schema but this bubbled up all the way in Spark and caused job
failures.
I just checked and SolrException class is actually in the consumer job jar
we use. Is there any reason why Spark cannot find the
This could be because org.apache.solr.common.SolrException doesn't
implement Serializable.
This error shows up when Spark is deserilizing a class which doesn't
implement Serializable.
Thanks
Divya
On Sep 29, 2015 4:37 PM, "Dmitry Goldenberg"
wrote:
> We're seeing this
I'm actually not sure how either one of these would possibly cause Spark to
find SolrException. Whether the driver or executor class path is first,
should it not matter, if the class is in the consumer job jar?
On Tue, Sep 29, 2015 at 9:12 PM, Dmitry Goldenberg
Ted, I think I have tried these settings with the hbase protocol jar, to no
avail.
I'm going to see if I can try and use these with this SolrException issue
though it now may be harder to reproduce it. Thanks for the suggestion.
On Tue, Sep 29, 2015 at 8:03 PM, Ted Yu