[GitHub] [spark] AmplabJenkins commented on issue #25308: [SPARK-28576][SQL] fix the dead lock issue when enable new adaptive execution

[GitBox]

AmplabJenkins commented on issue #25308: [SPARK-28576][SQL] fix the dead lock 
issue when enable new adaptive execution
URL: https://github.com/apache/spark/pull/25308#issuecomment-516708182
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25308: [SPARK-28576][SQL] fix the dead lock issue when enable new adaptive execution

[GitBox]

AmplabJenkins commented on issue #25308: [SPARK-28576][SQL] fix the dead lock 
issue when enable new adaptive execution
URL: https://github.com/apache/spark/pull/25308#issuecomment-516708050
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] JkSelf commented on issue #25308: [SPARK-28576][SQL] fix the dead lock issue when enable new adaptive execution

[GitBox]

JkSelf commented on issue #25308: [SPARK-28576][SQL] fix the dead lock issue 
when enable new adaptive execution
URL: https://github.com/apache/spark/pull/25308#issuecomment-516707537
 
 
   @maryannxue @carsonwang  Please help me review If you have available time. 
Thanks for your help.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25308: [SPARK-28576][SQL] fix the dead lock issue when enable new adaptive execution

[GitBox]

AmplabJenkins commented on issue #25308: [SPARK-28576][SQL] fix the dead lock 
issue when enable new adaptive execution
URL: https://github.com/apache/spark/pull/25308#issuecomment-516706936
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] JkSelf opened a new pull request #25308: [SPARK-28576][SQL] fix the dead lock issue when enable new adaptive execution

[GitBox]

JkSelf opened a new pull request #25308: [SPARK-28576][SQL] fix the dead lock 
issue when enable new adaptive execution
URL: https://github.com/apache/spark/pull/25308
 
 
   ## What changes were proposed in this pull request?
   In our local test 1TB TPC-DS, Q6 is hang when enable new AE. The issue is 
same with [PR#80](https://github.com/Intel-bigdata/spark-adaptive/pull/80). The 
root cause is that when execute the subquery the 
[synchronized](https://github.com/apache/spark/blob/a745381b9d3dd290057ef3089de7fdb9264f1f8b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L245)
 and [canonicalized lazy 
operator](https://github.com/apache/spark/blob/a745381b9d3dd290057ef3089de7fdb9264f1f8b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L213)
 are all lock the same FilterExec. And this PR change the synchronized to 
ReentrantLock to fix the hang issue.
   
   ## How was this patch tested?
   Manual test Q6 1TB TPC-DS


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] sarutak commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

sarutak commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be 
applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516705480
 
 
   It's strange. I use bash of same version for you.
   
   ```
   scala> :quit
   $ bash --version
   GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
   Copyright (C) 2007 Free Software Foundation, Inc.
   ```
   
   But the version of macOS is different. Is this the cause?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] sarutak edited a comment on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

sarutak edited a comment on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher 
to be applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516705480
 
 
   It's strange. I use bash of same version for you.
   
   ```
   $ bash --version
   GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
   Copyright (C) 2007 Free Software Foundation, Inc.
   ```
   
   But the version of macOS is different. Is this the cause?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] 
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r309043735
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
 ##
 @@ -532,6 +532,28 @@ class SparkListenerSuite extends SparkFunSuite with 
LocalSparkContext with Match
 }
   }
 
+  test("event queue size can be configued through spark conf") {
+val conf = new SparkConf(false)
+  .set(LISTENER_BUS_EVENT_QUEUE_CAPACITY, 5)
+  .set("spark.scheduler.listenerbus.eventqueue.shared.capacity", "1")
 
 Review comment:
   nit: I would use s`${SHARED_QUEUE}.capacity`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] 
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r309044216
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
 ##
 @@ -532,6 +532,28 @@ class SparkListenerSuite extends SparkFunSuite with 
LocalSparkContext with Match
 }
   }
 
+  test("event queue size can be configued through spark conf") {
+val conf = new SparkConf(false)
+  .set(LISTENER_BUS_EVENT_QUEUE_CAPACITY, 5)
+  .set("spark.scheduler.listenerbus.eventqueue.shared.capacity", "1")
+  .set("spark.scheduler.listenerbus.eventqueue.eventLog.capacity", "2")
+
+val bus = new LiveListenerBus(conf)
+val counter1 = new BasicJobCounter()
+val counter2 = new BasicJobCounter()
+val counter3 = new BasicJobCounter()
+
+bus.addToSharedQueue(counter1)
 
 Review comment:
   Maybe add comment to explain this is just to trigger add a new Queue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] 
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r309043848
 
 

 ##
 File path: 
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
 ##
 @@ -532,6 +532,28 @@ class SparkListenerSuite extends SparkFunSuite with 
LocalSparkContext with Match
 }
   }
 
+  test("event queue size can be configued through spark conf") {
+val conf = new SparkConf(false)
+  .set(LISTENER_BUS_EVENT_QUEUE_CAPACITY, 5)
+  .set("spark.scheduler.listenerbus.eventqueue.shared.capacity", "1")
+  .set("spark.scheduler.listenerbus.eventqueue.eventLog.capacity", "2")
 
 Review comment:
   similarly, I would use s`${EVENT_LOG_QUEUE}.capacity`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] 
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r309044457
 
 

 ##
 File path: core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
 ##
 @@ -46,8 +46,13 @@ private class AsyncEventQueue(
 
   // Cap the capacity of the queue so we get an explicit error (rather than an 
OOM exception) if
   // it's perpetually being added to more quickly than it's being drained.
-  private val eventQueue = new LinkedBlockingQueue[SparkListenerEvent](
-conf.get(LISTENER_BUS_EVENT_QUEUE_CAPACITY))
+  // The capacity can be configured by 
spark.scheduler.listenerbus.eventqueue.${name}.capacity,
+  // if no such conf is specified, use the value specified in
+  // LISTENER_BUS_EVENT_QUEUE_CAPACITY
+  protected def capacity: Int = conf.getInt(
+s"spark.scheduler.listenerbus.eventqueue.${name}.capacity",
 
 Review comment:
   We shall assert the capacity is > 0, and add test to cover the case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on a change in pull request #25307: [SPARK-28574][CORE] 
Allow to config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#discussion_r309045345
 
 

 ##
 File path: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala
 ##
 @@ -236,6 +236,13 @@ private[spark] class LiveListenerBus(conf: SparkConf) {
 queues.asScala.map(_.name).toSet
   }
 
+  // For testing only.
+  private[scheduler] def getQueueCapacity(name: String): Int = {
+queues.asScala.find(_.name == name) match {
 
 Review comment:
   maybe `queues.asScala.find(_.name == name).map(_.capacity).getOrElse(-1)` ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] 
Return null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309043580
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -456,9 +474,52 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
-case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (longValue == longValue.toInt) {
+  longValue.toInt
+} else {
+  null
+}
+  })
+case ByteType =>
+  b => b.asInstanceOf[Byte].toInt
+case ShortType =>
+  b => b.asInstanceOf[Short].toInt
+case LongType =>
+  buildCast[Long](_, l =>
+if (l == l.toInt) {
+  l.toInt
+} else {
+  null
+}
+  )
+case x: FloatType =>
+  buildCast[Float](_, f =>
+if (f <= Int.MaxValue && f >= Int.MinValue) {
+  f.toInt
 
 Review comment:
   Actually it is tricty to compare float and int if the float value is around 
Int.MaxValue or Int.MinValue.
   ```
   scala> BigDecimal((Int.MaxValue + 1L).toString).toFloat <= Int.MaxValue
   res1: Boolean = true
   ```
   This is because `float` is also 32 bits long, and it uses 8 bit to represent 
the exponent field. While `int` is 32 bits long. So the comparison won't be 
accurate.We don't have to worry about the rounding for Float -> Int.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

SparkQA commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should 
respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516702199
 
 
   **[Test build #108445 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108445/testReport)**
 for PR 25002 at commit 
[`612dadb`](https://github.com/apache/spark/commit/612dadbf25a3f273a38bc68f7bc35832dc0a6744).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

HyukjinKwon closed pull request #25245: [SPARK-25382][SQL][PYSPARK] Remove 
ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

AmplabJenkins removed a comment on issue #25002: [SPARK-28203][Core][Python] 
PythonRDD should respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516701789
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

AmplabJenkins removed a comment on issue #25002: [SPARK-28203][Core][Python] 
PythonRDD should respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516701793
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13541/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

AmplabJenkins commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD 
should respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516701793
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13541/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

AmplabJenkins commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD 
should respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516701789
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] 
Return null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309043580
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -456,9 +474,52 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
-case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (longValue == longValue.toInt) {
+  longValue.toInt
+} else {
+  null
+}
+  })
+case ByteType =>
+  b => b.asInstanceOf[Byte].toInt
+case ShortType =>
+  b => b.asInstanceOf[Short].toInt
+case LongType =>
+  buildCast[Long](_, l =>
+if (l == l.toInt) {
+  l.toInt
+} else {
+  null
+}
+  )
+case x: FloatType =>
+  buildCast[Float](_, f =>
+if (f <= Int.MaxValue && f >= Int.MinValue) {
+  f.toInt
 
 Review comment:
   Actually it is tricty to compare float and int if the float value is around 
Int.MaxValue or Int.MinValue.
   ```
   scala> BigDecimal((Int.MaxValue + 1L).toString).toFloat <= Int.MaxValue
   res1: Boolean = true
   ```
   This is because `float` is also 32 bits long, and it uses 8 bit to represent 
the exponent field. While `int` is 32 bits long. So the comparison won't be 
accurate.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

HyukjinKwon commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove 
ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516701139
 
 
   Merged to master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

HyukjinKwon commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD 
should respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516701088
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25305: [SPARK-28572] Simple analyzer checks for CREATE TABLE v2

[GitBox]

AmplabJenkins removed a comment on issue #25305: [SPARK-28572] Simple analyzer 
checks for CREATE TABLE v2
URL: https://github.com/apache/spark/pull/25305#issuecomment-516700017
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108428/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25305: [SPARK-28572] Simple analyzer checks for CREATE TABLE v2

[GitBox]

AmplabJenkins removed a comment on issue #25305: [SPARK-28572] Simple analyzer 
checks for CREATE TABLE v2
URL: https://github.com/apache/spark/pull/25305#issuecomment-516700014
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25305: [SPARK-28572] Simple analyzer checks for CREATE TABLE v2

[GitBox]

AmplabJenkins commented on issue #25305: [SPARK-28572] Simple analyzer checks 
for CREATE TABLE v2
URL: https://github.com/apache/spark/pull/25305#issuecomment-516700014
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25305: [SPARK-28572] Simple analyzer checks for CREATE TABLE v2

[GitBox]

AmplabJenkins commented on issue #25305: [SPARK-28572] Simple analyzer checks 
for CREATE TABLE v2
URL: https://github.com/apache/spark/pull/25305#issuecomment-516700017
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108428/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on issue #25307: [SPARK-28574][CORE] Allow to config 
different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516699867
 
 
   ok to test


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

AmplabJenkins removed a comment on issue #25307: [SPARK-28574][CORE] Allow to 
config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516691502
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25305: [SPARK-28572] Simple analyzer checks for CREATE TABLE v2

[GitBox]

SparkQA removed a comment on issue #25305: [SPARK-28572] Simple analyzer checks 
for CREATE TABLE v2
URL: https://github.com/apache/spark/pull/25305#issuecomment-516653140
 
 
   **[Test build #108428 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108428/testReport)**
 for PR 25305 at commit 
[`93ce1e8`](https://github.com/apache/spark/commit/93ce1e802224fe126acfc3d2e5dfd90c8f18bc47).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25305: [SPARK-28572] Simple analyzer checks for CREATE TABLE v2

[GitBox]

SparkQA commented on issue #25305: [SPARK-28572] Simple analyzer checks for 
CREATE TABLE v2
URL: https://github.com/apache/spark/pull/25305#issuecomment-516699629
 
 
   **[Test build #108428 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108428/testReport)**
 for PR 25305 at commit 
[`93ce1e8`](https://github.com/apache/spark/commit/93ce1e802224fe126acfc3d2e5dfd90c8f18bc47).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `case class UnresolvedCatalogRelation(tableMeta: CatalogTable) extends 
LeafNode `


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jiangxb1987 commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

jiangxb1987 commented on issue #25307: [SPARK-28574][CORE] Allow to config 
different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516699495
 
 
   add to whitelist


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

[GitBox]

AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of 
different dialects insensitive to case by JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-516698291
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

[GitBox]

AmplabJenkins commented on issue #25287: [SPARK-28552][SQL]Identification of 
different dialects insensitive to case by JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-516698296
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108426/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

[GitBox]

AmplabJenkins removed a comment on issue #25287: 
[SPARK-28552][SQL]Identification of different dialects insensitive to case by 
JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-516698291
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

[GitBox]

AmplabJenkins removed a comment on issue #25287: 
[SPARK-28552][SQL]Identification of different dialects insensitive to case by 
JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-516698296
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108426/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null 
result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516698075
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108440/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null 
result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516698071
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result 
on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516698071
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result 
on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516698075
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108440/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

SparkQA removed a comment on issue #25300: [SPARK-28503][SQL] Return null 
result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516677419
 
 
   **[Test build #108440 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108440/testReport)**
 for PR 25300 at commit 
[`e62551a`](https://github.com/apache/spark/commit/e62551ab8d70844799aa891c2f457758541a2816).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

SparkQA commented on issue #25300: [SPARK-28503][SQL] Return null result on 
cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516697988
 
 
   **[Test build #108440 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108440/testReport)**
 for PR 25300 at commit 
[`e62551a`](https://github.com/apache/spark/commit/e62551ab8d70844799aa891c2f457758541a2816).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

[GitBox]

SparkQA removed a comment on issue #25287: [SPARK-28552][SQL]Identification of 
different dialects insensitive to case by JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-516646433
 
 
   **[Test build #108426 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108426/testReport)**
 for PR 25287 at commit 
[`02f5cf9`](https://github.com/apache/spark/commit/02f5cf999bb35b536f48200d42e01f2e119a9b09).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

[GitBox]

SparkQA commented on issue #25287: [SPARK-28552][SQL]Identification of 
different dialects insensitive to case by JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-516697842
 
 
   **[Test build #108426 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108426/testReport)**
 for PR 25287 at commit 
[`02f5cf9`](https://github.com/apache/spark/commit/02f5cf999bb35b536f48200d42e01f2e119a9b09).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return 
null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309038260
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -456,9 +474,52 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
-case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (longValue == longValue.toInt) {
 
 Review comment:
   oh, right. I misunderstood it. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null 
result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516697187
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108439/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

SparkQA removed a comment on issue #25300: [SPARK-28503][SQL] Return null 
result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516675993
 
 
   **[Test build #108439 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108439/testReport)**
 for PR 25300 at commit 
[`b504f2e`](https://github.com/apache/spark/commit/b504f2e375e15c4d5d48036731e78b27).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result 
on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516697183
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins commented on issue #25300: [SPARK-28503][SQL] Return null result 
on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516697187
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108439/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

AmplabJenkins removed a comment on issue #25300: [SPARK-28503][SQL] Return null 
result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516697183
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

SparkQA commented on issue #25300: [SPARK-28503][SQL] Return null result on 
cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516697073
 
 
   **[Test build #108439 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108439/testReport)**
 for PR 25300 at commit 
[`b504f2e`](https://github.com/apache/spark/commit/b504f2e375e15c4d5d48036731e78b27).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] advancedxy commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD should respect SparkContext's hadoop configuration

[GitBox]

advancedxy commented on issue #25002: [SPARK-28203][Core][Python] PythonRDD 
should respect SparkContext's hadoop configuration
URL: https://github.com/apache/spark/pull/25002#issuecomment-516696566
 
 
   @HyukjinKwon The failure looks unrelated to this commit, could you trigger 
the test again or add
   me to the whitelist?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

maropu commented on issue #25300: [SPARK-28503][SQL] Return null result on cast 
an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#issuecomment-516695674
 
 
   I've left the same comment in 
https://github.com/apache/spark/pull/25239#discussion_r307068104 though, `Cast` 
itself needs to support this kind of upcast behaivours? As another option, we 
could check value ranges by `IF` expressions: 
https://github.com/apache/spark/compare/master...maropu:SPARK-28503


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return 
null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309038260
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -456,9 +474,52 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
-case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (longValue == longValue.toInt) {
 
 Review comment:
   oh, yeah. I missed it. Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] 
Return null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309037729
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -456,9 +474,52 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
-case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (longValue == longValue.toInt) {
 
 Review comment:
   Do you mean: is it possible that`longValue == longValue.toInt` when 
longValue can't fit into `Int`? 
   The `longValue` is long type, so it is impossible.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: 
OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#issuecomment-516692717
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13540/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

AmplabJenkins removed a comment on issue #25172: [SPARK-28412][SQL] ANSI SQL: 
OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#issuecomment-516692711
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY 
function support byte array
URL: https://github.com/apache/spark/pull/25172#issuecomment-516692717
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13540/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

AmplabJenkins commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY 
function support byte array
URL: https://github.com/apache/spark/pull/25172#issuecomment-516692711
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

gengliangwang commented on a change in pull request #25300: [SPARK-28503][SQL] 
Return null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309036523
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -180,13 +180,14 @@ object Cast {
 
 case (FloatType | DoubleType, TimestampType) => true
 case (TimestampType, DateType) => false
+case (TimestampType, _: IntegralType) if to != LongType => true
 
 Review comment:
   Yes, converting TimeStamp to Byte/Short/Int type can be null


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

[GitBox]

SparkQA commented on issue #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY 
function support byte array
URL: https://github.com/apache/spark/pull/25172#issuecomment-516691601
 
 
   **[Test build #108444 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108444/testReport)**
 for PR 25172 at commit 
[`dcca443`](https://github.com/apache/spark/commit/dcca44361a87df7462e73bdff023896336c2fedf).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

AmplabJenkins commented on issue #25307: [SPARK-28574][CORE] Allow to config 
different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516691502
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

AmplabJenkins removed a comment on issue #25307: [SPARK-28574][CORE] Allow to 
config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516690991
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

AmplabJenkins removed a comment on issue #25307: [SPARK-28574][CORE] Allow to 
config different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516690858
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

AmplabJenkins commented on issue #25307: [SPARK-28574][CORE] Allow to config 
different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516690991
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25307: [SPARK-28574][CORE] Allow to config different sizes for event queues

[GitBox]

AmplabJenkins commented on issue #25307: [SPARK-28574][CORE] Allow to config 
different sizes for event queues
URL: https://github.com/apache/spark/pull/25307#issuecomment-516690858
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] yunzoud opened a new pull request #25307: [SPARK-28574] Allow to config different sizes for event queues

[GitBox]

yunzoud opened a new pull request #25307: [SPARK-28574] Allow to config 
different sizes for event queues
URL: https://github.com/apache/spark/pull/25307
 
 
   ## What changes were proposed in this pull request?
   Add configuration spark.scheduler.listenerbus.eventqueue.${name}.capacity to 
allow configuration of different event queue size.
   
   ## How was this patch tested?
   Unit test in 
core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] 
Remove ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516690339
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108434/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

AmplabJenkins removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] 
Remove ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516690337
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove 
ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516690339
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108434/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

AmplabJenkins commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove 
ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516690337
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on 
arithmetic operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-516689609
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13539/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

SparkQA commented on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove 
ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516689963
 
 
   **[Test build #108434 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108434/testReport)**
 for PR 25245 at commit 
[`6460c7a`](https://github.com/apache/spark/commit/6460c7a33d2ff9c61e63117ac6362469a834a9e3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on 
arithmetic operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-516689604
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

[GitBox]

SparkQA removed a comment on issue #25245: [SPARK-25382][SQL][PYSPARK] Remove 
ImageSchema.readImages in 3.0
URL: https://github.com/apache/spark/pull/25245#issuecomment-516660718
 
 
   **[Test build #108434 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108434/testReport)**
 for PR 25245 at commit 
[`6460c7a`](https://github.com/apache/spark/commit/6460c7a33d2ff9c61e63117ac6362469a834a9e3).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on 
arithmetic operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-516689609
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13539/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on 
arithmetic operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-516689604
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

maropu commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be 
applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516689506
 
 
   It seems my bash is too old;
   ```
   $bash --version
   GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)
   Copyright (C) 2007 Free Software Foundation, Inc.
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

[GitBox]

AmplabJenkins removed a comment on issue #25279: [WIP][SPARK-28519][SQL] Use 
StrictMath log, pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-516689048
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108430/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

[GitBox]

AmplabJenkins removed a comment on issue #25279: [WIP][SPARK-28519][SQL] Use 
StrictMath log, pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-516689044
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

[GitBox]

SparkQA removed a comment on issue #25279: [WIP][SPARK-28519][SQL] Use 
StrictMath log, pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-516655938
 
 
   **[Test build #108430 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108430/testReport)**
 for PR 25279 at commit 
[`c5ca3ea`](https://github.com/apache/spark/commit/c5ca3ea1875c8643268180f79d18264e047c8499).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

[GitBox]

AmplabJenkins commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath 
log, pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-516689044
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

[GitBox]

AmplabJenkins commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath 
log, pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-516689048
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108430/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

[GitBox]

SparkQA commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, 
pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-516688828
 
 
   **[Test build #108430 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108430/testReport)**
 for PR 25279 at commit 
[`c5ca3ea`](https://github.com/apache/spark/commit/c5ca3ea1875c8643268180f79d18264e047c8499).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `case class Exp(child: Expression) extends 
UnaryMathExpression(StrictMath.exp, \"EXP\") `
 * `case class Expm1(child: Expression) extends 
UnaryMathExpression(StrictMath.expm1, \"EXPM1\") `


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] kiszk edited a comment on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

kiszk edited a comment on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to 
be applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516686052
 
 
   In my environment (Ubuntu 18.04), it works well.
   
   ```
   $ /usr/bin/env bash --version
   GNU bash, version 4.4.19(1)-release (powerpc64le-unknown-linux-gnu)
   Copyright (C) 2016 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later 
   
   This is free software; you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.
   
   $ git log | head -1
   commit dba4375359a2dfed1f009edc3b1bcf6b3253fe02
   $ ./bin/spark-shell 
   19/07/31 13:01:44 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
   Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
   Setting default log level to "WARN".
   To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use 
setLogLevel(newLevel).
   Spark context Web UI available at http://foo.trl.ibm.com:4040
   Spark context available as 'sc' (master = local[*], app id = 
local-1564545717751).
   Spark session available as 'spark'.
   Welcome to
   __
/ __/__  ___ _/ /__
   _\ \/ _ \/ _ `/ __/  '_/
  /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-SNAPSHOT
 /_/

   Using Scala version 2.12.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
   Type in expressions to have them evaluated.
   Type :help for more information.
   
   scala>
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

SparkQA commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic 
operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-516688558
 
 
   **[Test build #108443 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108443/testReport)**
 for PR 21599 at commit 
[`b809a3f`](https://github.com/apache/spark/commit/b809a3fadd1acbb4d5f061f46a370ffc90348beb).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

[GitBox]

maropu commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic 
operations returns incorrect result
URL: https://github.com/apache/spark/pull/21599#issuecomment-516688375
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return 
null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309032828
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -442,8 +452,16 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
   buildCast[Int](_, d => null)
 case TimestampType =>
   buildCast[Long](_, t => timestampToLong(t))
+case ByteType =>
+  b => b.asInstanceOf[Byte].toLong
+case ShortType =>
+  b => b.asInstanceOf[Short].toLong
+case IntegerType =>
+  b => b.asInstanceOf[Int].toLong
+case _: DecimalType =>
+  b => castDecimalToLong(b.asInstanceOf[Decimal])
 case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toLong(b)
+  b => castDecimalToLong(Decimal(b.toString))
 
 Review comment:
   I'm a bit worried about string conversion overheads here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

maropu commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be 
applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516686596
 
 
   oh, I see. I'll check again later, thx!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] kiszk commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

kiszk commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be 
applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516686052
 
 
   In my environment (Ubuntu 18.04), it works well.
   
   ```
   $ git log | head -1
   commit dba4375359a2dfed1f009edc3b1bcf6b3253fe02
   $  ./bin/spark-shell 
   19/07/31 13:01:44 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
   Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
   Setting default log level to "WARN".
   To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use 
setLogLevel(newLevel).
   Spark context Web UI available at http://s7fp1.trl.ibm.com:4040
   Spark context available as 'sc' (master = local[*], app id = 
local-1564545717751).
   Spark session available as 'spark'.
   Welcome to
   __
/ __/__  ___ _/ /__
   _\ \/ _ \/ _ `/ __/  '_/
  /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-SNAPSHOT
 /_/

   Using Scala version 2.12.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
   Type in expressions to have them evaluated.
   Type :help for more information.
   
   scala>
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] kiszk edited a comment on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

kiszk edited a comment on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to 
be applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516686052
 
 
   In my environment (Ubuntu 18.04), it works well.
   
   ```
   $ git log | head -1
   commit dba4375359a2dfed1f009edc3b1bcf6b3253fe02
   $  ./bin/spark-shell 
   19/07/31 13:01:44 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
   Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
   Setting default log level to "WARN".
   To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use 
setLogLevel(newLevel).
   Spark context Web UI available at http://foo.trl.ibm.com:4040
   Spark context available as 'sc' (master = local[*], app id = 
local-1564545717751).
   Spark session available as 'spark'.
   Welcome to
   __
/ __/__  ___ _/ /__
   _\ \/ _ \/ _ `/ __/  '_/
  /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-SNAPSHOT
 /_/

   Using Scala version 2.12.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
   Type in expressions to have them evaluated.
   Type :help for more information.
   
   scala>
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return 
null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309030466
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -180,13 +180,14 @@ object Cast {
 
 case (FloatType | DoubleType, TimestampType) => true
 case (TimestampType, DateType) => false
+case (TimestampType, _: IntegralType) if to != LongType => true
 
 Review comment:
   Is this related to this pr?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

[GitBox]

maropu commented on a change in pull request #25300: [SPARK-28503][SQL] Return 
null result on cast an out-of-range value to a integral type
URL: https://github.com/apache/spark/pull/25300#discussion_r309029995
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ##
 @@ -456,9 +474,52 @@ case class Cast(child: Expression, dataType: DataType, 
timeZoneId: Option[String
 case DateType =>
   buildCast[Int](_, d => null)
 case TimestampType =>
-  buildCast[Long](_, t => timestampToLong(t).toInt)
-case x: NumericType =>
-  b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b)
+  buildCast[Long](_, t => {
+val longValue = timestampToLong(t)
+if (longValue == longValue.toInt) {
+  longValue.toInt
+} else {
+  null
+}
+  })
+case ByteType =>
+  b => b.asInstanceOf[Byte].toInt
+case ShortType =>
+  b => b.asInstanceOf[Short].toInt
+case LongType =>
+  buildCast[Long](_, l =>
+if (l == l.toInt) {
+  l.toInt
+} else {
+  null
+}
+  )
+case x: FloatType =>
+  buildCast[Float](_, f =>
+if (f <= Int.MaxValue && f >= Int.MinValue) {
+  f.toInt
 
 Review comment:
   This question might be out-of-scope in this pr though... we don't need 
round-up here?
   ```
   postgres=# select CAST(3.9 AS INT);
int4 
   --
   4
   (1 row)
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

[GitBox]

maropu commented on issue #25265: [SPARK-28525][DEPLOY] Allow Launcher to be 
applied Java options
URL: https://github.com/apache/spark/pull/25265#issuecomment-516683236
 
 
   I fetched this commit and my spark-shell seemed to be broken... do I miss 
something?
   ```
   (py27) maropu@~/Repositories/spark/spark-master: (master 
$%>)$./bin/spark-shell
   NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes 
ahead of assembly.
   /Users/maropu/Repositories/spark/spark-master/bin/spark-class: line 96: CMD: 
bad array subscript
   head: illegal line count -- -1
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-516681735
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

AmplabJenkins removed a comment on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-516682672
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-516682788
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-516682672
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] advancedxy commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

advancedxy commented on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-516681888
 
 
   cc @cloud-fan 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] advancedxy commented on a change in pull request #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

advancedxy commented on a change in pull request #25306: [SPARK-28573][SQL] 
Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for 
partitioned table
URL: https://github.com/apache/spark/pull/25306#discussion_r309027655
 
 

 ##
 File path: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
 ##
 @@ -197,7 +197,9 @@ private[hive] class HiveMetastoreCatalog(sparkSession: 
SparkSession) extends Log
   Some(partitionSchema))
 
 val logicalRelation = cached.getOrElse {
-  val sizeInBytes = relation.stats.sizeInBytes.toLong
+  val defaultSizeInBytes = 
sparkSession.sessionState.conf.defaultSizeInBytes
 
 Review comment:
   For write path, the stats for HiveTable is not calculated as 
InsertIntoTable's table filed is not added in the children.
   
   So `org.apache.spark.sql.hive.DetermineTableStats` skip this table


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

[GitBox]

AmplabJenkins commented on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-516681735
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org