Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
+1, thanks @sarutak
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes s
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@tgravescs No problem. Have a nice vacation :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
@soldiershen did you try this patch rather then putting the hdfs conf file
on the hosts?
@sarutak sorry been out on vacation and going on vacation again, probably
won't get to this til n
Github user soldiershen commented on the issue:
https://github.com/apache/spark/pull/13738
@sarutak got it. I add hdfs conf file to specific the host .Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@soldiershen Doesn't this patch address the issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user soldiershen commented on the issue:
https://github.com/apache/spark/pull/13738
Hi guys,I met same issue(spark 1.6.0) when using "val rdd1 =
hiveContext.sql("select SUBSTR(startjointime,1,10)...
println(rdd1.collect().toString)"
but it's normal when using hiveContext.s
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62819/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #62819 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62819/consoleFull)**
for PR 13738 at commit
[`fca2b4a`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #62819 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62819/consoleFull)**
for PR 13738 at commit
[`fca2b4a`](https://github.com/apache/spark/commit/f
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
Sorry for the late response. I'll try it the way you mentioned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
Sorry for my delay on this, so it looks like if we create any FileSystem is
will cause the service loaders to load all services, so in hadoopRDD,
newApiHadoopRDD where we don't know the path we co
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
Yeah, I agree that `spark-csv` case is the special one and not a matter of
Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
you should file a bug with databricks to fix spark-csv. sorry haven't had
a chance to get back to this, I'll look more today.
I would prefer to only load it when you are doing something h
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
In Spark 2.0, this issue cannot happen when we use the dataframe load
method because as you mentioned, all of file-based datasource does a
`hdfsPath.getFileSystem`.
I noticed there is at le
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
does this happen when you use the dataframe load method? I'm guessing not
because the datasource code does a hdfsPath.getFileSystem.
for hadoopFile, textFile, newApiHadoopFile since we h
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
thanks for the explanation, I understand now.
ok on the loading it looks like
https://issues.apache.org/jira/browse/SPARK-8135 broke this because its not
loading the defaults anymore. Th
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@tgravescs I reproduced this with following condition.
(1) Made `spark-default.conf` empty
(2) Only `HADOOP_CONF_DIR=/path/to/hadoop-conf` in spark-env.sh
(3) NameNode HA is enabled and se
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
and on the second question, what command was used to reproduce this exactly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
@tgravescs I didn't debug against 1.4 but I think the reason why 1.5+ can
throw UnknownHostException while 1.4 works successfully is related to the
difference between `SerializableWritable` and `Se
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/13738
the jira says this worked in spark 1.4 but not spark 1.5 or 1.6, do we know
why? was spark loading the configuration somewhere else early on in spark 1.4?
I must be misunderstanding your
Github user sarutak commented on the issue:
https://github.com/apache/spark/pull/13738
Maybe anyone of @rxin , @srowen or @tgravescs could review this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13738
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60711/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/13738
**[Test build #60711 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60711/consoleFull)**
for PR 13738 at commit
[`d41bfe0`](https://github.com/apache/spark/commit/
25 matches
Mail list logo
