[jira] [Commented] (SPARK-27541) Refresh class definitions for jars added via addJar()
[ https://issues.apache.org/jira/browse/SPARK-27541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831408#comment-16831408 ] Chakravarthi commented on SPARK-27541: -- [~navedalam] I would like to look into this > Refresh class definitions for jars added via addJar() > - > > Key: SPARK-27541 > URL: https://issues.apache.org/jira/browse/SPARK-27541 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.2.3 >Reporter: Naved Alam >Priority: Major > > Currently, if a class is loaded by the executor, its definition cannot be > updated (because classloaders won't load an already loaded class again). For > use cases with long running sparkContexts, this becomes a problem when there > are requirements to update the definition of one of these classes. > There should be a spark property which when turned on, allows the executors > to refresh the definitions of these classes if they were to be added as a new > jar using the addJar API. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27543) Support getRequiredJars and getRequiredFiles APIs for Hive UDFs
[ https://issues.apache.org/jira/browse/SPARK-27543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831407#comment-16831407 ] Chakravarthi commented on SPARK-27543: -- [~makagonov] SparkContext.addJar could be used to include the required additional resources . > Support getRequiredJars and getRequiredFiles APIs for Hive UDFs > --- > > Key: SPARK-27543 > URL: https://issues.apache.org/jira/browse/SPARK-27543 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.0.0, 2.4.1 >Reporter: Sergey >Priority: Minor > Original Estimate: 1,344h > Remaining Estimate: 1,344h > > *getRequiredJars* and *getRequiredFiles* - functions to automatically include > additional resources required by a UDF. The files that are provided in > methods would be accessible by executors by simple file name. This is > necessary for UDFs that need to have some required files distributed, or > classes from third-party jars to be available from executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27336) Incorrect DataSet.summary() result
[
https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806947#comment-16806947
]
Chakravarthi commented on SPARK-27336:
--
I'm checking this issue.
> Incorrect DataSet.summary() result
> --
>
> Key: SPARK-27336
> URL: https://issues.apache.org/jira/browse/SPARK-27336
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Gengliang Wang
>Priority: Major
> Attachments: test.csv
>
>
> There is a single data point in the minimum_nights column that is 1.0E8 out
> of 8k records, but .summary() says it is the 75% and the max.
> I compared this with approxQuantile, and approxQuantile for 75% gave the
> correct value of 30.0.
> To reproduce:
> {code:java}
> scala> val df =
> spark.read.format("csv").load("test.csv").withColumn("minimum_nights",
> '_c0.cast("Int"))
> df: org.apache.spark.sql.DataFrame = [_c0: string, minimum_nights: int]
> scala> df.select("minimum_nights").summary().show()
> +---+--+
> |summary|minimum_nights|
> +---+--+
> | count| 7072|
> | mean| 14156.35407239819|
> | stddev|1189128.5444975856|
> |min| 1|
> |25%| 2|
> |50%| 4|
> |75%| 1|
> |max| 1|
> +---+--+
> scala> df.stat.approxQuantile("minimum_nights", Array(0.75), 0.1)
> res1: Array[Double] = Array(30.0)
> scala> df.stat.approxQuantile("minimum_nights", Array(0.75), 0.001)
> res2: Array[Double] = Array(30.0)
> scala> df.stat.approxQuantile("minimum_nights", Array(0.75), 0.0001)
> res3: Array[Double] = Array(1.0E8)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27324) document configurations related to executor metrics
[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806225#comment-16806225 ] Chakravarthi commented on SPARK-27324: -- Thanks for reporting,will submit PR for this. > document configurations related to executor metrics > --- > > Key: SPARK-27324 > URL: https://issues.apache.org/jira/browse/SPARK-27324 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Wing Yew Poon >Priority: Major > > SPARK-23429 introduced executor memory metrics, and the configuration, > spark.eventLog.logStageExecutorMetrics.enabled, that determines if per-stage > per-executor metric peaks get written to the event log. (The metrics are > polled and sent in the heartbeat, and this is always done; the configuration > is only to determine if aggregated metric peaks are written to the event log.) > SPARK-24958 added proc fs based metrics to the executor memory metrics, and > the configuration, spark.eventLog.logStageExecutorProcessTreeMetrics.enabled, > to determine if these additional (more expensive) metrics are collected when > metrics are polled. > SPARK-26329 will introduce a configuration, > spark.executor.metrics.pollingInterval, to allow polling at more frequent > intervals than the executor heartbeat. > These configurations and how they relate to each other should be documented > in the Configuration page. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27255) Aggregate functions should not be allowed in WHERE
[
https://issues.apache.org/jira/browse/SPARK-27255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800972#comment-16800972
]
Chakravarthi commented on SPARK-27255:
--
Hi [~dkbiswal] , sorry for the delay...I just started analyzing it,you can
submit your PR .
> Aggregate functions should not be allowed in WHERE
> --
>
> Key: SPARK-27255
> URL: https://issues.apache.org/jira/browse/SPARK-27255
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Mingcong Han
>Priority: Minor
>
> Aggregate functions should not be allowed in WHERE clause. But Spark SQL
> throws an exception when generating codes. It is supposed to throw an
> exception during parsing or analyzing.
> Here is an example:
> {code:scala}
> val df = spark.sql("select * from t where sum(ta) > 0")
> df.explain(true)
> df.show()
> {code}
> Spark SQL explains it as:
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'Filter ('sum('ta) > 0)
>+- 'UnresolvedRelation `t`
> == Analyzed Logical Plan ==
> ta: int, tb: int
> Project [ta#5, tb#6]
> +- Filter (sum(cast(ta#5 as bigint)) > cast(0 as bigint))
>+- SubqueryAlias `t`
> +- Project [ta#5, tb#6]
> +- SubqueryAlias `as`
> +- LocalRelation [ta#5, tb#6]
> == Optimized Logical Plan ==
> Filter (sum(cast(ta#5 as bigint)) > 0)
> +- LocalRelation [ta#5, tb#6]
> == Physical Plan ==
> *(1) Filter (sum(cast(ta#5 as bigint)) > 0)
> +- LocalTableScan [ta#5, tb#6]
> {noformat}
> But when executing `df.show()`:
> {noformat}
> Exception in thread "main" java.lang.UnsupportedOperationException: Cannot
> generate code for expression: sum(cast(input[0, int, false] as bigint))
> at
> org.apache.spark.sql.catalyst.expressions.Unevaluable.doGenCode(Expression.scala:291)
> at
> org.apache.spark.sql.catalyst.expressions.Unevaluable.doGenCode$(Expression.scala:290)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.doGenCode(interfaces.scala:87)
> at
> org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:138)
> at scala.Option.getOrElse(Option.scala:138)
> {noformat}
> I have tried it in PostgreSQL, and it directly throws an error:
> {noformat}
> ERROR: Aggregate functions are not allowed in WHERE.
> {noformat}
> We'd better throw an AnalysisException here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27255) Aggregate functions should not be allowed in WHERE
[
https://issues.apache.org/jira/browse/SPARK-27255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799671#comment-16799671
]
Chakravarthi commented on SPARK-27255:
--
Thanks for reporting,will be working on this issue.
> Aggregate functions should not be allowed in WHERE
> --
>
> Key: SPARK-27255
> URL: https://issues.apache.org/jira/browse/SPARK-27255
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Mingcong Han
>Priority: Minor
>
> Aggregate functions should not be allowed in WHERE clause. But Spark SQL
> throws an exception when generating codes. It is supposed to throw an
> exception during parsing or analyzing.
> Here is an example:
> {code:scala}
> val df = spark.sql("select * from t where sum(ta) > 0")
> df.explain(true)
> df.show()
> {code}
> Spark SQL explains it as:
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'Filter ('sum('ta) > 0)
>+- 'UnresolvedRelation `t`
> == Analyzed Logical Plan ==
> ta: int, tb: int
> Project [ta#5, tb#6]
> +- Filter (sum(cast(ta#5 as bigint)) > cast(0 as bigint))
>+- SubqueryAlias `t`
> +- Project [ta#5, tb#6]
> +- SubqueryAlias `as`
> +- LocalRelation [ta#5, tb#6]
> == Optimized Logical Plan ==
> Filter (sum(cast(ta#5 as bigint)) > 0)
> +- LocalRelation [ta#5, tb#6]
> == Physical Plan ==
> *(1) Filter (sum(cast(ta#5 as bigint)) > 0)
> +- LocalTableScan [ta#5, tb#6]
> {noformat}
> But when executing `df.show()`:
> {noformat}
> Exception in thread "main" java.lang.UnsupportedOperationException: Cannot
> generate code for expression: sum(cast(input[0, int, false] as bigint))
> at
> org.apache.spark.sql.catalyst.expressions.Unevaluable.doGenCode(Expression.scala:291)
> at
> org.apache.spark.sql.catalyst.expressions.Unevaluable.doGenCode$(Expression.scala:290)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression.doGenCode(interfaces.scala:87)
> at
> org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:138)
> at scala.Option.getOrElse(Option.scala:138)
> {noformat}
> I have tried it in PostgreSQL, and it directly throws an error:
> {noformat}
> ERROR: Aggregate functions are not allowed in WHERE.
> {noformat}
> We'd better throw an AnalysisException here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27253) SparkSession clone discards SQLConf overrides in favor of SparkConf defaults
[ https://issues.apache.org/jira/browse/SPARK-27253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799574#comment-16799574 ] Chakravarthi commented on SPARK-27253: -- Thanks for reporting ,will be working on this . > SparkSession clone discards SQLConf overrides in favor of SparkConf defaults > > > Key: SPARK-27253 > URL: https://issues.apache.org/jira/browse/SPARK-27253 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.4.0 >Reporter: Jose Torres >Priority: Major > > SparkSession.cloneSession() is normally supposed to create a child session > which inherits all the SQLConf values of its parent session. But when a SQL > conf is given a global default through the SparkConf, this does not happen; > the child session will receive the SparkConf default rather than its parent's > SQLConf override. > > This is particularly impactful in structured streaming, as the microbatches > run in a cloned child session. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799543#comment-16799543 ] Chakravarthi commented on SPARK-27244: -- [~needna78] ok > Conf Log Displays Password In Clear Text > > > Key: SPARK-27244 > URL: https://issues.apache.org/jira/browse/SPARK-27244 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.3.2, 2.3.3, 2.4.0 >Reporter: Ninad >Priority: Major > > When logConf is set to true and any sensitive information like passwords is > set using spark-submit command then the passwords are logged in clear text > from conf. Ideally, these passwords should be redacted as it does on UI and > then printed to log for debugging purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799124#comment-16799124 ] Chakravarthi commented on SPARK-27244: -- [~kabhwan] working on this,will submit a PR soon. > Conf Log Displays Password In Clear Text > > > Key: SPARK-27244 > URL: https://issues.apache.org/jira/browse/SPARK-27244 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.3.2, 2.3.3, 2.4.0 >Reporter: Ninad >Priority: Major > > When logConf is set to true and any sensitive information like passwords is > set using spark-submit command then the passwords are logged in clear text > from conf. Ideally, these passwords should be redacted as it does on UI and > then printed to log for debugging purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798949#comment-16798949 ] Chakravarthi commented on SPARK-27244: -- I will try to analyze this. > Conf Log Displays Password In Clear Text > > > Key: SPARK-27244 > URL: https://issues.apache.org/jira/browse/SPARK-27244 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.3.2, 2.3.3, 2.4.0 >Reporter: Ninad >Priority: Major > > When logConf is set to true and any sensitive information like passwords is > set using spark-submit command then the passwords are logged in clear text > from conf. Ideally, these passwords should be redacted as it does on UI and > then printed to log for debugging purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27246) scalar subquery with no columns throws exception in spark.shell
Chakravarthi created SPARK-27246:
Summary: scalar subquery with no columns throws exception in
spark.shell
Key: SPARK-27246
URL: https://issues.apache.org/jira/browse/SPARK-27246
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.2
Reporter: Chakravarthi
scala> val exp = ScalarSubquery(LocalRelation()).as('a)
java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
at scala.collection.IterableLike$class.head(IterableLike.scala:107)
at
scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:186)
at
scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:126)
at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:186)
at
org.apache.spark.sql.catalyst.expressions.ScalarSubquery.dataType(subquery.scala:251)
at
org.apache.spark.sql.catalyst.expressions.Alias.dataType(namedExpressions.scala:147)
at
org.apache.spark.sql.catalyst.expressions.NamedExpression$class.typeSuffix(namedExpressions.scala:89)
at
org.apache.spark.sql.catalyst.expressions.Alias.typeSuffix(namedExpressions.scala:129)
at
org.apache.spark.sql.catalyst.expressions.Alias.toString(namedExpressions.scala:176)
at
scala.runtime.ScalaRunTime$.scala$runtime$ScalaRunTime$$inner$1(ScalaRunTime.scala:332)
at scala.runtime.ScalaRunTime$.stringOf(ScalaRunTime.scala:337)
at scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345)
at .$print$lzycompute(:10)
at .$print(:6)
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at
scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at
scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at
scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:415)
at
scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:923)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at
scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:76)
at org.apache.spark.repl.Main$.main(Main.scala:56)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:925)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:201)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:231)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:140)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql
[
https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791364#comment-16791364
]
Chakravarthi commented on SPARK-27017:
--
Hi, [~uNxe] sorry for delay, please check this jira : SPARK-21912
> Creating orc table with special symbols in column name via spark.sql
>
>
> Key: SPARK-27017
> URL: https://issues.apache.org/jira/browse/SPARK-27017
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
>Affects Versions: 2.3.0
>Reporter: Henryk Cesnolovic
>Priority: Major
>
> Issue is creating orc table with special symbols in column name in spark with
> hive support. Example:
> _spark.sql("Create table abc_orc (`Column with speci@l symbo|s`string) stored
> as orc")_
> throws org.apache.spark.sql.AnalysisException: Column name "Column with
> speci@l symbo|s" contains invalid character(s). Please use alias to rename it.
> It's interesting, because in Hive we can create such table and after that in
> spark we can select data from that table and it resolves schema correctly.
> My question is, is it correct behaviour of spark and if so, what is the
> reason of that behaviour?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27088) Apply conf "spark.sql.optimizer.planChangeLog.level" to batch plan change in RuleExecutor
[ https://issues.apache.org/jira/browse/SPARK-27088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789170#comment-16789170 ] Chakravarthi commented on SPARK-27088: -- [~maryannxue] will be working on this > Apply conf "spark.sql.optimizer.planChangeLog.level" to batch plan change in > RuleExecutor > - > > Key: SPARK-27088 > URL: https://issues.apache.org/jira/browse/SPARK-27088 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.0 >Reporter: Maryann Xue >Priority: Minor > > Similar to SPARK-25415, which has made log level for plan changes by each > rule configurable, we can make log level for plan changes by each batch > configurable too and can reuse the same configuration: > "spark.sql.optimizer.planChangeLog.level". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql
[
https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786789#comment-16786789
]
Chakravarthi commented on SPARK-27017:
--
[~uNxe] could you provide queries you have used,because Hive also does not
allow special character in column name .
> Creating orc table with special symbols in column name via spark.sql
>
>
> Key: SPARK-27017
> URL: https://issues.apache.org/jira/browse/SPARK-27017
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
>Affects Versions: 2.3.0
>Reporter: Henryk Cesnolovic
>Priority: Major
>
> Issue is creating orc table with special symbols in column name in spark with
> hive support. Example:
> _spark.sql("Create table abc_orc (`Column with speci@l symbo|s`string) stored
> as orc")_
> throws org.apache.spark.sql.AnalysisException: Column name "Column with
> speci@l symbo|s" contains invalid character(s). Please use alias to rename it.
> It's interesting, because in Hive we can create such table and after that in
> spark we can select data from that table and it resolves schema correctly.
> My question is, is it correct behaviour of spark and if so, what is the
> reason of that behaviour?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27072) Changing the parameter value of completedJob.sort to X prints stacktrace in sparkWebUI
[ https://issues.apache.org/jira/browse/SPARK-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786781#comment-16786781 ] Chakravarthi commented on SPARK-27072: -- [~vanzin] Thanks for your answer. > Changing the parameter value of completedJob.sort to X prints stacktrace in > sparkWebUI > -- > > Key: SPARK-27072 > URL: https://issues.apache.org/jira/browse/SPARK-27072 > Project: Spark > Issue Type: Question > Components: Web UI >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > > Manipulating the value of completedJob.sort parameter > From > x.x.x.x:4040/jobs/?=Description=100#completed > To > x.x.x.x:4040/jobs/job/?id=1=x > is printing Stacktrace in webUI > > java.lang.IllegalArgumentException: Unknown column: x at > org.apache.spark.ui.jobs.JobDataSource.ordering(AllJobsPage.scala:493) at > org.apache.spark.ui.jobs.JobDataSource.(AllJobsPage.scala:441) at > org.apache.spark.ui.jobs.JobPagedTable.(AllJobsPage.scala:533) at > org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:248) at > org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:297) at > org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at > org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at > org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at > javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at > javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at > org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > at > org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) > at > org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > at > org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > at > org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > at > org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) > at > org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > at > org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > at org.spark_project.jetty.server.Server.handle(Server.java:539) at > org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at > org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > at > org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) > at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108) > at > org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > at java.lang.Thread.run(Thread.java:748) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26879) Inconsistency in default column names for functions like inline and stack
[
https://issues.apache.org/jira/browse/SPARK-26879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786401#comment-16786401
]
Chakravarthi commented on SPARK-26879:
--
Thanks for reporting. [~jashgala] I would like to work on this issue.
> Inconsistency in default column names for functions like inline and stack
> -
>
> Key: SPARK-26879
> URL: https://issues.apache.org/jira/browse/SPARK-26879
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Jash Gala
>Priority: Minor
>
> In the Spark SQL functions definitions, `inline` uses col1, col2, etc. (i.e.
> 1-indexed columns), while `stack` uses col0, col1, col2, etc. (i.e. 0-indexed
> columns).
> {code:title=spark-shell|borderStyle=solid}
> scala> spark.sql("SELECT stack(2, 1, 2, 3)").show
> +++
> |col0|col1|
> +++
> | 1| 2|
> | 3|null|
> +++
> scala> spark.sql("SELECT inline_outer(array(struct(1, 'a'), struct(2,
> 'b')))").show
> +++
> |col1|col2|
> +++
> | 1| a|
> | 2| b|
> +++
> {code}
> This feels like an issue with consistency. As discussed on [PR
> #23748|https://github.com/apache/spark/pull/23748], it might be a good idea
> to standardize this to something specific (like zero-based indexing) for
> these and other similar functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27072) Changing the parameter value of completedJob.sort to X prints stacktrace in sparkWebUI
[ https://issues.apache.org/jira/browse/SPARK-27072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785713#comment-16785713 ] Chakravarthi commented on SPARK-27072: -- Hi [~vanzin] , Please confirm, is it ok to print stacktrace in the UI?. If wrong column name is appended to URL,we can print like "Invalid column name" instead of throwing exception with stacktrace right? > Changing the parameter value of completedJob.sort to X prints stacktrace in > sparkWebUI > -- > > Key: SPARK-27072 > URL: https://issues.apache.org/jira/browse/SPARK-27072 > Project: Spark > Issue Type: Question > Components: Web UI >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > > Manipulating the value of completedJob.sort parameter > From > x.x.x.x:4040/jobs/?=Description=100#completed > To > x.x.x.x:4040/jobs/job/?id=1=x > is printing Stacktrace in webUI > > java.lang.IllegalArgumentException: Unknown column: x at > org.apache.spark.ui.jobs.JobDataSource.ordering(AllJobsPage.scala:493) at > org.apache.spark.ui.jobs.JobDataSource.(AllJobsPage.scala:441) at > org.apache.spark.ui.jobs.JobPagedTable.(AllJobsPage.scala:533) at > org.apache.spark.ui.jobs.AllJobsPage.jobsTable(AllJobsPage.scala:248) at > org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:297) at > org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at > org.apache.spark.ui.WebUI$$anonfun$2.apply(WebUI.scala:84) at > org.apache.spark.ui.JettyUtils$$anon$3.doGet(JettyUtils.scala:90) at > javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at > javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at > org.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > at > org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) > at > org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > at > org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > at > org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > at > org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) > at > org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) > at > org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) > at org.spark_project.jetty.server.Server.handle(Server.java:539) at > org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at > org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) > at > org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) > at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108) > at > org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) > at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) > at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) > at > org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) > at > org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) > at > org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) > at java.lang.Thread.run(Thread.java:748) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql
[
https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785640#comment-16785640
]
Chakravarthi commented on SPARK-27017:
--
Hi [~uNxe] ,can you please provide which hive version you are using ?
> Creating orc table with special symbols in column name via spark.sql
>
>
> Key: SPARK-27017
> URL: https://issues.apache.org/jira/browse/SPARK-27017
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
>Affects Versions: 2.3.0
>Reporter: Henryk Cesnolovic
>Priority: Major
>
> Issue is creating orc table with special symbols in column name in spark with
> hive support. Example:
> _spark.sql("Create table abc_orc (`Column with speci@l symbo|s`string) stored
> as orc")_
> throws org.apache.spark.sql.AnalysisException: Column name "Column with
> speci@l symbo|s" contains invalid character(s). Please use alias to rename it.
> It's interesting, because in Hive we can create such table and after that in
> spark we can select data from that table and it resolves schema correctly.
> My question is, is it correct behaviour of spark and if so, what is the
> reason of that behaviour?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26602) Subsequent queries are failing after querying the UDF which is loaded with wrong hdfs path
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26602: - Summary: Subsequent queries are failing after querying the UDF which is loaded with wrong hdfs path (was: Insert into table fails after querying the UDF which is loaded with wrong hdfs path) > Subsequent queries are failing after querying the UDF which is loaded with > wrong hdfs path > -- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > Attachments: beforeFixUdf.txt > > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784526#comment-16784526 ] Chakravarthi commented on SPARK-26602: -- [~srowen] agree,but it should not make any other subsequent query (at least query which does not refer that UDF) to fail right? . Any insert or select on the existing table itself is failing. [~ajithshetty] Yes,it makes all the subsequent query to fail,not only the query which refers to that UDF. > Insert into table fails after querying the UDF which is loaded with wrong > hdfs path > --- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > Attachments: beforeFixUdf.txt > > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path
[
https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784134#comment-16784134
]
Chakravarthi edited comment on SPARK-26602 at 3/5/19 2:31 PM:
--
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue
is,Insert into table fails after querying the UDF which is loaded with wrong
hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table table1(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external
resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table or select on any table.It will fail.
sql("insert into table1 values(1)").show
sql("select * from table1 ").show
Here ,insert should work.but is fails.
was (Author: chakravarthi):
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue
is,Insert into table fails after querying the UDF which is loaded with wrong
hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table table1(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external
resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table.
sql("insert into table1 values(1)").show
Here ,insert should work.but is fails.
> Insert into table fails after querying the UDF which is loaded with wrong
> hdfs path
> ---
>
> Key: SPARK-26602
> URL: https://issues.apache.org/jira/browse/SPARK-26602
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Haripriya
>Priority: Major
> Attachments: beforeFixUdf.txt
>
>
> In sql,
> 1.Query the existing udf(say myFunc1)
> 2. create and select the udf registered with incorrect path (say myFunc2)
> 3.Now again query the existing udf in the same session - Wil throw exception
> stating that couldn't read resource of myFunc2's path
> 4.Even the basic operations like insert and select will fail giving the same
> error
> Result:
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///tmp/hari_notexists1/two_udfs.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149)
> at
> org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841)
> at
> org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path
[
https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784134#comment-16784134
]
Chakravarthi edited comment on SPARK-26602 at 3/5/19 1:52 PM:
--
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue
is,Insert into table fails after querying the UDF which is loaded with wrong
hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table table1(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external
resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table.
sql("insert into table1 values(1)").show
Here ,insert should work.but is fails.
was (Author: chakravarthi):
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue
is,Insert into table fails after querying the UDF which is loaded with wrong
hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table check_udf(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external
resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table.
sql("insert into check_udf values(1)").show
Here ,insert should work.but is fails.
> Insert into table fails after querying the UDF which is loaded with wrong
> hdfs path
> ---
>
> Key: SPARK-26602
> URL: https://issues.apache.org/jira/browse/SPARK-26602
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Haripriya
>Priority: Major
> Attachments: beforeFixUdf.txt
>
>
> In sql,
> 1.Query the existing udf(say myFunc1)
> 2. create and select the udf registered with incorrect path (say myFunc2)
> 3.Now again query the existing udf in the same session - Wil throw exception
> stating that couldn't read resource of myFunc2's path
> 4.Even the basic operations like insert and select will fail giving the same
> error
> Result:
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///tmp/hari_notexists1/two_udfs.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149)
> at
> org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841)
> at
> org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26602: - Attachment: beforeFixUdf.txt > Insert into table fails after querying the UDF which is loaded with wrong > hdfs path > --- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > Attachments: beforeFixUdf.txt > > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi reopened SPARK-26602: -- > Insert into table fails after querying the UDF which is loaded with wrong > hdfs path > --- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26602: - Summary: Insert into table fails after querying the UDF which is loaded with wrong hdfs path (was: Once creating and quering udf with incorrect path,followed by querying tables or functions registered with correct path gives the runtime exception within the same session) > Insert into table fails after querying the UDF which is loaded with wrong > hdfs path > --- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26602) Once creating and quering udf with incorrect path,followed by querying tables or functions registered with correct path gives the runtime exception within the same ses
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784137#comment-16784137 ] Chakravarthi commented on SPARK-26602: -- And the problem is, though the jar does not exist ,it is added to the addedJars in sparkContext.scala, when performing select on UDF. So, when insert into table happens it is trying to load the jars from the ListJars and as the jar not exist,it gives Exception. The fix is to validate the Jar exist or not before adding to the addedJars. I have fixed it and will raise MR. > Once creating and quering udf with incorrect path,followed by querying tables > or functions registered with correct path gives the runtime exception within > the same session > --- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26602) Once creating and quering udf with incorrect path,followed by querying tables or functions registered with correct path gives the runtime exception within the same ses
[
https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784134#comment-16784134
]
Chakravarthi commented on SPARK-26602:
--
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue
is,Insert into table fails after querying the UDF which is loaded with wrong
hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table check_udf(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR
'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external
resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table.
sql("insert into check_udf values(1)").show
Here ,insert should work.but is fails.
> Once creating and quering udf with incorrect path,followed by querying tables
> or functions registered with correct path gives the runtime exception within
> the same session
> ---
>
> Key: SPARK-26602
> URL: https://issues.apache.org/jira/browse/SPARK-26602
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0
>Reporter: Haripriya
>Priority: Major
>
> In sql,
> 1.Query the existing udf(say myFunc1)
> 2. create and select the udf registered with incorrect path (say myFunc2)
> 3.Now again query the existing udf in the same session - Wil throw exception
> stating that couldn't read resource of myFunc2's path
> 4.Even the basic operations like insert and select will fail giving the same
> error
> Result:
> java.lang.RuntimeException: Failed to read external resource
> hdfs:///tmp/hari_notexists1/two_udfs.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288)
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149)
> at
> org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841)
> at
> org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27017) Creating orc table with special symbols in column name via spark.sql
[
https://issues.apache.org/jira/browse/SPARK-27017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781346#comment-16781346
]
Chakravarthi commented on SPARK-27017:
--
I'm checking this issue.
> Creating orc table with special symbols in column name via spark.sql
>
>
> Key: SPARK-27017
> URL: https://issues.apache.org/jira/browse/SPARK-27017
> Project: Spark
> Issue Type: Question
> Components: Spark Shell
>Affects Versions: 2.3.0
>Reporter: Henryk Cesnolovic
>Priority: Major
>
> Issue is creating orc table with special symbols in column name in spark with
> hive support. Example:
> _spark.sql("Create table abc_orc (`Column with speci@l symbo|s`string) stored
> as orc")_
> throws org.apache.spark.sql.AnalysisException: Column name "Column with
> speci@l symbo|s" contains invalid character(s). Please use alias to rename it.
> It's interesting, because in Hive we can create such table and after that in
> spark we can select data from that table and it resolves schema correctly.
> My question is, is it correct behaviour of spark and if so, what is the
> reason of that behaviour?
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26602) Once creating and quering udf with incorrect path,followed by querying tables or functions registered with correct path gives the runtime exception within the same ses
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1695#comment-1695 ] Chakravarthi commented on SPARK-26602: -- I will be working on this issue. > Once creating and quering udf with incorrect path,followed by querying tables > or functions registered with correct path gives the runtime exception within > the same session > --- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.0 >Reporter: Haripriya >Priority: Major > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26255: - Attachment: command.png > Custom error/exception is not thrown for the SQL tab when UI filters are > added in spark-sql launch > -- > > Key: SPARK-26255 > URL: https://issues.apache.org/jira/browse/SPARK-26255 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.3.2 > Environment: 【Test Environment】： > Server OS :-SUSE > No. of Cluster Node:-3 > Spark Version:- 2.3.2 > Hadoop Version:-3.1 >Reporter: Sushanta Sen >Priority: Major > Attachments: command.png, logs_before_fix.png, ui_befofre_fix.png > > > 【Detailed description】：Custom error is not thrown for the SQL tab when UI > filters are added in spark-sql launch > 【Precondition】： > 1.Cluster is up and running【Test step】： > 1. Launch spark sql as below: > [spark-sql --master yarn --conf > spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter > --conf > spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple"] > 2. Go to Yarn application list UI link > 3. Launch the application master for the Spark-SQL app ID > 4. It will display an error > 5. Append /executors, /stages, /jobs, /environment, /SQL > 【Expect Output】：An error should be displayed "An error has occurred. Please > check for all the TABS > 【Actual Output】：The error message is displayed for all the tabs except SQL > tab . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720353#comment-16720353 ] Chakravarthi edited comment on SPARK-26255 at 12/13/18 4:16 PM: [~hyukjin.kwon] I have attached the snapshots was (Author: chakravarthi): [~hyukjin.kwon]I have attached the snapshots > Custom error/exception is not thrown for the SQL tab when UI filters are > added in spark-sql launch > -- > > Key: SPARK-26255 > URL: https://issues.apache.org/jira/browse/SPARK-26255 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.3.2 > Environment: 【Test Environment】： > Server OS :-SUSE > No. of Cluster Node:-3 > Spark Version:- 2.3.2 > Hadoop Version:-3.1 >Reporter: Sushanta Sen >Priority: Major > Attachments: command.png, logs_before_fix.png, ui_befofre_fix.png > > > 【Detailed description】：Custom error is not thrown for the SQL tab when UI > filters are added in spark-sql launch > 【Precondition】： > 1.Cluster is up and running【Test step】： > 1. Launch spark sql as below: > [spark-sql --master yarn --conf > spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter > --conf > spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple"] > 2. Go to Yarn application list UI link > 3. Launch the application master for the Spark-SQL app ID > 4. It will display an error > 5. Append /executors, /stages, /jobs, /environment, /SQL > 【Expect Output】：An error should be displayed "An error has occurred. Please > check for all the TABS > 【Actual Output】：The error message is displayed for all the tabs except SQL > tab . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720353#comment-16720353 ] Chakravarthi commented on SPARK-26255: -- [~hyukjin.kwon]I have attached the snapshots > Custom error/exception is not thrown for the SQL tab when UI filters are > added in spark-sql launch > -- > > Key: SPARK-26255 > URL: https://issues.apache.org/jira/browse/SPARK-26255 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.3.2 > Environment: 【Test Environment】： > Server OS :-SUSE > No. of Cluster Node:-3 > Spark Version:- 2.3.2 > Hadoop Version:-3.1 >Reporter: Sushanta Sen >Priority: Major > Attachments: command.png, logs_before_fix.png, ui_befofre_fix.png > > > 【Detailed description】：Custom error is not thrown for the SQL tab when UI > filters are added in spark-sql launch > 【Precondition】： > 1.Cluster is up and running【Test step】： > 1. Launch spark sql as below: > [spark-sql --master yarn --conf > spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter > --conf > spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple"] > 2. Go to Yarn application list UI link > 3. Launch the application master for the Spark-SQL app ID > 4. It will display an error > 5. Append /executors, /stages, /jobs, /environment, /SQL > 【Expect Output】：An error should be displayed "An error has occurred. Please > check for all the TABS > 【Actual Output】：The error message is displayed for all the tabs except SQL > tab . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26255: - Attachment: logs_before_fix.png > Custom error/exception is not thrown for the SQL tab when UI filters are > added in spark-sql launch > -- > > Key: SPARK-26255 > URL: https://issues.apache.org/jira/browse/SPARK-26255 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.3.2 > Environment: 【Test Environment】： > Server OS :-SUSE > No. of Cluster Node:-3 > Spark Version:- 2.3.2 > Hadoop Version:-3.1 >Reporter: Sushanta Sen >Priority: Major > Attachments: logs_before_fix.png, ui_befofre_fix.png > > > 【Detailed description】：Custom error is not thrown for the SQL tab when UI > filters are added in spark-sql launch > 【Precondition】： > 1.Cluster is up and running【Test step】： > 1. Launch spark sql as below: > [spark-sql --master yarn --conf > spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter > --conf > spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple"] > 2. Go to Yarn application list UI link > 3. Launch the application master for the Spark-SQL app ID > 4. It will display an error > 5. Append /executors, /stages, /jobs, /environment, /SQL > 【Expect Output】：An error should be displayed "An error has occurred. Please > check for all the TABS > 【Actual Output】：The error message is displayed for all the tabs except SQL > tab . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26255: - Attachment: ui_befofre_fix.png > Custom error/exception is not thrown for the SQL tab when UI filters are > added in spark-sql launch > -- > > Key: SPARK-26255 > URL: https://issues.apache.org/jira/browse/SPARK-26255 > Project: Spark > Issue Type: Bug > Components: SQL, Web UI >Affects Versions: 2.3.2 > Environment: 【Test Environment】： > Server OS :-SUSE > No. of Cluster Node:-3 > Spark Version:- 2.3.2 > Hadoop Version:-3.1 >Reporter: Sushanta Sen >Priority: Major > Attachments: logs_before_fix.png, ui_befofre_fix.png > > > 【Detailed description】：Custom error is not thrown for the SQL tab when UI > filters are added in spark-sql launch > 【Precondition】： > 1.Cluster is up and running【Test step】： > 1. Launch spark sql as below: > [spark-sql --master yarn --conf > spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter > --conf > spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple"] > 2. Go to Yarn application list UI link > 3. Launch the application master for the Spark-SQL app ID > 4. It will display an error > 5. Append /executors, /stages, /jobs, /environment, /SQL > 【Expect Output】：An error should be displayed "An error has occurred. Please > check for all the TABS > 【Actual Output】：The error message is displayed for all the tabs except SQL > tab . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26255) Custom error/exception is not thrown for the SQL tab when UI filters are added in spark-sql launch
[ https://issues.apache.org/jira/browse/SPARK-26255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706989#comment-16706989 ] Chakravarthi commented on SPARK-26255: -- I will be working on this issue. > Custom error/exception is not thrown for the SQL tab when UI filters are > added in spark-sql launch > -- > > Key: SPARK-26255 > URL: https://issues.apache.org/jira/browse/SPARK-26255 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.2 > Environment: 【Test Environment】： > Server OS :-SUSE > No. of Cluster Node:-3 > Spark Version:- 2.3.2 > Hadoop Version:-3.1 >Reporter: Sushanta Sen >Priority: Major > > 【Detailed description】：Custom error is not thrown for the SQL tab when UI > filters are added in spark-sql launch > 【Precondition】： > 1.Cluster is up and running【Test step】： > 1. Launch spark sql as below: > spark-sql --master yarn --conf > spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter > --conf > spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params="type=simple" > 2. Go to Yarn application list UI link > 3. Launch the application master for the Spark-SQL app ID > 4. It will display an error > 5. Append /executors, /stages, /jobs, /environment, /SQL > 【Expect Output】：An error should be displayed "An error has occurred. Please > check for all the TABS > 【Actual Output】：The error message is displayed for all the tabs except SQL > tab . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
