[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17501675#comment-17501675 ] Ismael Juma commented on SPARK-25075: - Yeah, I was asking since Kafka plans to drop support for Scala 2.12 next year https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=181308218#content/view/181308218 > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Assignee: Sean R. Owen >Priority: Major > Fix For: 3.2.0 > > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498639#comment-17498639 ] Ismael Juma commented on SPARK-25075: - Is there a very rough timeline for 4.0 or it completely unknown at this stage? > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Assignee: Sean R. Owen >Priority: Major > Fix For: 3.2.0 > > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17498639#comment-17498639 ] Ismael Juma edited comment on SPARK-25075 at 2/27/22, 5:39 PM: --- Is there a very rough timeline for 4.0 or is it completely unknown at this stage? was (Author: ijuma): Is there a very rough timeline for 4.0 or it completely unknown at this stage? > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Assignee: Sean R. Owen >Priority: Major > Fix For: 3.2.0 > > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36808) Upgrade Kafka to 2.8.1
[ https://issues.apache.org/jira/browse/SPARK-36808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492248#comment-17492248 ] Ismael Juma commented on SPARK-36808: - I think it would be good to backport this to the stable version of Spark as it includes a number of important fixes: [https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html] > Upgrade Kafka to 2.8.1 > -- > > Key: SPARK-36808 > URL: https://issues.apache.org/jira/browse/SPARK-36808 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0 > > > A few hours ago, Kafka 2.8.1 was released, which includes a bunch of bug fix. > https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33913) Upgrade Kafka to 2.8.0
[ https://issues.apache.org/jira/browse/SPARK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492247#comment-17492247 ] Ismael Juma commented on SPARK-33913: - Sounds good, I had not seen the other ticket. > Upgrade Kafka to 2.8.0 > -- > > Key: SPARK-33913 > URL: https://issues.apache.org/jira/browse/SPARK-33913 > Project: Spark > Issue Type: Sub-task > Components: Build, DStreams >Affects Versions: 3.2.0 >Reporter: dengziming >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.2.0 > > > This issue aims to upgrade Kafka client to 2.8.0. > Note that Kafka 2.8.0 uses ZSTD JNI 1.4.9-1 like Apache Spark 3.2.0. > *RELEASE NOTE* > - https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html > - https://downloads.apache.org/kafka/2.7.0/RELEASE_NOTES.html > This will bring the latest client-side improvement and bug fixes like the > following examples. > - KAFKA-10631 ProducerFencedException is not Handled on Offest Commit > - KAFKA-10134 High CPU issue during rebalance in Kafka consumer after > upgrading to 2.5 > - KAFKA-12193 Re-resolve IPs when a client is disconnected > - KAFKA-10090 Misleading warnings: The configuration was supplied but isn't a > known config > - KAFKA-9263 The new hw is added to incorrect log when > ReplicaAlterLogDirsThread is replacing log > - KAFKA-10607 Ensure the error counts contains the NONE > - KAFKA-10458 Need a way to update quota for TokenBucket registered with > Sensor > - KAFKA-10503 MockProducer doesn't throw ClassCastException when no partition > for topic -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-33913) Upgrade Kafka to 2.8.0
[ https://issues.apache.org/jira/browse/SPARK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491535#comment-17491535 ] Ismael Juma edited comment on SPARK-33913 at 2/13/22, 9:17 AM: --- Note that it may be a good idea to upgrade the stable branch of spark to include Kafka 2.8.1 as it includes a number of fixes: [https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html] was (Author: ijuma): Note that it may be a good idea to upgrade the stable branch of spark to include Kafka 2.8.1 as it includes a number of fixed: https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html > Upgrade Kafka to 2.8.0 > -- > > Key: SPARK-33913 > URL: https://issues.apache.org/jira/browse/SPARK-33913 > Project: Spark > Issue Type: Sub-task > Components: Build, DStreams >Affects Versions: 3.2.0 >Reporter: dengziming >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.2.0 > > > This issue aims to upgrade Kafka client to 2.8.0. > Note that Kafka 2.8.0 uses ZSTD JNI 1.4.9-1 like Apache Spark 3.2.0. > *RELEASE NOTE* > - https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html > - https://downloads.apache.org/kafka/2.7.0/RELEASE_NOTES.html > This will bring the latest client-side improvement and bug fixes like the > following examples. > - KAFKA-10631 ProducerFencedException is not Handled on Offest Commit > - KAFKA-10134 High CPU issue during rebalance in Kafka consumer after > upgrading to 2.5 > - KAFKA-12193 Re-resolve IPs when a client is disconnected > - KAFKA-10090 Misleading warnings: The configuration was supplied but isn't a > known config > - KAFKA-9263 The new hw is added to incorrect log when > ReplicaAlterLogDirsThread is replacing log > - KAFKA-10607 Ensure the error counts contains the NONE > - KAFKA-10458 Need a way to update quota for TokenBucket registered with > Sensor > - KAFKA-10503 MockProducer doesn't throw ClassCastException when no partition > for topic -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33913) Upgrade Kafka to 2.8.0
[ https://issues.apache.org/jira/browse/SPARK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17491535#comment-17491535 ] Ismael Juma commented on SPARK-33913: - Note that it may be a good idea to upgrade the stable branch of spark to include Kafka 2.8.1 as it includes a number of fixed: https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html > Upgrade Kafka to 2.8.0 > -- > > Key: SPARK-33913 > URL: https://issues.apache.org/jira/browse/SPARK-33913 > Project: Spark > Issue Type: Sub-task > Components: Build, DStreams >Affects Versions: 3.2.0 >Reporter: dengziming >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.2.0 > > > This issue aims to upgrade Kafka client to 2.8.0. > Note that Kafka 2.8.0 uses ZSTD JNI 1.4.9-1 like Apache Spark 3.2.0. > *RELEASE NOTE* > - https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html > - https://downloads.apache.org/kafka/2.7.0/RELEASE_NOTES.html > This will bring the latest client-side improvement and bug fixes like the > following examples. > - KAFKA-10631 ProducerFencedException is not Handled on Offest Commit > - KAFKA-10134 High CPU issue during rebalance in Kafka consumer after > upgrading to 2.5 > - KAFKA-12193 Re-resolve IPs when a client is disconnected > - KAFKA-10090 Misleading warnings: The configuration was supplied but isn't a > known config > - KAFKA-9263 The new hw is added to incorrect log when > ReplicaAlterLogDirsThread is replacing log > - KAFKA-10607 Ensure the error counts contains the NONE > - KAFKA-10458 Need a way to update quota for TokenBucket registered with > Sensor > - KAFKA-10503 MockProducer doesn't throw ClassCastException when no partition > for topic -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350084#comment-17350084 ] Ismael Juma commented on SPARK-25075: - Yes, makes sense. When I mentioned Kafka 3.0 and Spark 3.2, I meant from the point of view of users that want to use both together. I understand that Spark itself will go with Kafka 2.8 for Spark 3.2 and evaluate the upgrade to Kafka 3.0 once it's available (expected sometime in July/August). > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350068#comment-17350068 ] Ismael Juma commented on SPARK-25075: - Thanks Dongjoon, I totally agree that stable Scala 2.13 support needs to come first. I was asking because we are also trying to figure out the strategy for Kafka when it comes to Scala versions support. We have supported Scala 2.13 for a while, but Scala 2.12 is still widely used due to Spark. I think I will propose to deprecate Scala 2.12 support in Apache Kafka 3.0, which would roughly align with Spark 3.2. And removal would happen around 1 year after that. > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17349932#comment-17349932 ] Ismael Juma commented on SPARK-25075: - [~dongjoon] When do you intend to drop support for Scala 2.12? > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28367) Kafka connector infinite wait because metadata never updated
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087242#comment-17087242 ] Ismael Juma commented on SPARK-28367: - This should be unblocked since Spark has upgraded to Kafka 2.5.0: [https://github.com/apache/spark/pull/28235] > Kafka connector infinite wait because metadata never updated > > > Key: SPARK-28367 > URL: https://issues.apache.org/jira/browse/SPARK-28367 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.1.3, 2.2.3, 2.3.3, 2.4.3, 3.0.0 >Reporter: Gabor Somogyi >Priority: Critical > > Spark uses an old and deprecated API named poll(long) which never returns and > stays in live lock if metadata is not updated (for instance when broker > disappears at consumer creation). > I've created a small standalone application to test it and the alternatives: > https://github.com/gaborgsomogyi/kafka-get-assignment -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31464) Upgrade Kafka to 2.5.0
[ https://issues.apache.org/jira/browse/SPARK-31464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087240#comment-17087240 ] Ismael Juma commented on SPARK-31464: - Thanks for upgrading so quickly. :) > Upgrade Kafka to 2.5.0 > -- > > Key: SPARK-31464 > URL: https://issues.apache.org/jira/browse/SPARK-31464 > Project: Spark > Issue Type: Improvement > Components: Build, Structured Streaming >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824226#comment-16824226 ] Ismael Juma commented on SPARK-25954: - That's awesome, sorry for the noise. > Upgrade to Kafka 2.1.0 > -- > > Key: SPARK-25954 > URL: https://issues.apache.org/jira/browse/SPARK-25954 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.0.0 > > > Kafka 2.1.0 vote passed. Since this includes official KAFKA-7264 JDK 11 > support, we had better use that. > - > https://lists.apache.org/thread.html/9f487094491e512b556a1c9c3c6034ac642b088e3f797e3d192ebc9d@%3Cdev.kafka.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824177#comment-16824177 ] Ismael Juma commented on SPARK-25954: - [~dongjoon] Kafka 2.2.0 has a fix for an issue that has caused problems for Spark in the past: https://cwiki.apache.org/confluence/display/KAFKA/KIP-207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+monotonically+increasing+even+during+a+partition+leader+change It might be worth upgrading when you have a chance. > Upgrade to Kafka 2.1.0 > -- > > Key: SPARK-25954 > URL: https://issues.apache.org/jira/browse/SPARK-25954 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.0.0 > > > Kafka 2.1.0 vote passed. Since this includes official KAFKA-7264 JDK 11 > support, we had better use that. > - > https://lists.apache.org/thread.html/9f487094491e512b556a1c9c3c6034ac642b088e3f797e3d192ebc9d@%3Cdev.kafka.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501351#comment-16501351 ] Ismael Juma commented on SPARK-18057: - Yes, it is [~kabhwan]. The major version bump is simply because support for Java 7 has been dropped. > Update structured streaming kafka from 0.10.0.1 to 1.1.0 > > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger >Priority: Major > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 0.10.0.1 to 1.1.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496924#comment-16496924 ] Ismael Juma commented on SPARK-18057: - Apache Kafka 2.0.0 will include KIP-266 and KAFKA-4879 has also been fixed. It would be great for Spark to transition to clients 2.0.0 once it's released. The code remains compatible, but Java 8 is now required. > Update structured streaming kafka from 0.10.0.1 to 1.1.0 > > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger >Priority: Major > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-1145) Memory mapping with many small blocks can cause JVM allocation failures
[
https://issues.apache.org/jira/browse/SPARK-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated SPARK-1145:
---
Comment: was deleted
(was: Thanks for the additional information. The issue seems similar to
SPARK-1145 where a large number of memory map operations eventually causes a OS
level failure.)
> Memory mapping with many small blocks can cause JVM allocation failures
> ---
>
> Key: SPARK-1145
> URL: https://issues.apache.org/jira/browse/SPARK-1145
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 0.9.0
>Reporter: Patrick Wendell
>Assignee: Patrick Wendell
> Fix For: 0.9.2, 1.0.0
>
>
> During a shuffle each block or block segment is memory mapped to a file. When
> the segments are very small and there are a large number of them, the memory
> maps can start failing and eventually the JVM will terminate. It's not clear
> exactly what's happening but it appears that when the JVM terminates about
> 265MB of virtual address space is used by memory mapped files. This doesn't
> seem affected at all by `-XXmaxdirectmemorysize` - AFAIK that option is just
> to give the JVM its own self imposed limit rather than allow it to run into
> OS limits.
> At the time of JVM failure it appears the overall OS memory becomes scarce,
> so it's possible there are overheads for each memory mapped file that are
> adding up here. One overhead is that the memory mapping occurs at the
> granularity of pages, so if blocks are really small there is natural overhead
> required to pad to the page boundary.
> In the particular case where I saw this, the JVM was running 4 reducers, each
> of which was trying to access about 30,000 blocks for a total of 120,000
> concurrent reads. At about 65,000 open files it crapped out. In this case
> each file was about 1000 bytes.
> User should really be coalescing or using fewer reducers if they have 1000
> byte shuffle files, but I expect this to happen nonetheless. My proposal was
> that if the file is smaller than a few pages, we should just read it into a
> java buffer and not bother to memory map it. Memory mapping huge numbers of
> small files in the JVM is neither recommended or good for performance, AFAIK.
> Below is the stack trace:
> {code}
> 14/02/27 08:32:35 ERROR storage.BlockManagerWorker: Exception handling buffer
> message
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
> at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:89)
> at
> org.apache.spark.storage.BlockManager.getLocalBytes(BlockManager.scala:285)
> at
> org.apache.spark.storage.BlockManagerWorker.getBlock(BlockManagerWorker.scala:90)
> at
> org.apache.spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:69)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at
> org.apache.spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:28)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at
> org.apache.spark.storage.BlockMessageArray.map(BlockMessageArray.scala:28)
> at
> org.apache.spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:44)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34)
> at
> org.apache.spark.network.ConnectionManager.org$apache$spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:512)
> at
> org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:478)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> {code}
> And the JVM error log had a bunch of entries like this:
> {code}
> 7f4b48f89000-7f4b48f8a000 r--s ca:30 1622077901
> /mnt4/spark/spark-local-20140227020022-227c/26/shuffle_0_22312_38
> 7f4b48f8a000-7f4b48f8b000 r--s ca:20 545892715
> /mnt3/spark/spark-local-20140227020022-5ef5/3a/shuffle_0_26808_20
> 7f4b48f8b000-7f4b48f8c000
[jira] [Commented] (SPARK-1145) Memory mapping with many small blocks can cause JVM allocation failures
[
https://issues.apache.org/jira/browse/SPARK-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285846#comment-16285846
]
Ismael Juma commented on SPARK-1145:
Thanks for the additional information. The issue seems similar to SPARK-1145
where a large number of memory map operations eventually causes a OS level
failure.
> Memory mapping with many small blocks can cause JVM allocation failures
> ---
>
> Key: SPARK-1145
> URL: https://issues.apache.org/jira/browse/SPARK-1145
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 0.9.0
>Reporter: Patrick Wendell
>Assignee: Patrick Wendell
> Fix For: 0.9.2, 1.0.0
>
>
> During a shuffle each block or block segment is memory mapped to a file. When
> the segments are very small and there are a large number of them, the memory
> maps can start failing and eventually the JVM will terminate. It's not clear
> exactly what's happening but it appears that when the JVM terminates about
> 265MB of virtual address space is used by memory mapped files. This doesn't
> seem affected at all by `-XXmaxdirectmemorysize` - AFAIK that option is just
> to give the JVM its own self imposed limit rather than allow it to run into
> OS limits.
> At the time of JVM failure it appears the overall OS memory becomes scarce,
> so it's possible there are overheads for each memory mapped file that are
> adding up here. One overhead is that the memory mapping occurs at the
> granularity of pages, so if blocks are really small there is natural overhead
> required to pad to the page boundary.
> In the particular case where I saw this, the JVM was running 4 reducers, each
> of which was trying to access about 30,000 blocks for a total of 120,000
> concurrent reads. At about 65,000 open files it crapped out. In this case
> each file was about 1000 bytes.
> User should really be coalescing or using fewer reducers if they have 1000
> byte shuffle files, but I expect this to happen nonetheless. My proposal was
> that if the file is smaller than a few pages, we should just read it into a
> java buffer and not bother to memory map it. Memory mapping huge numbers of
> small files in the JVM is neither recommended or good for performance, AFAIK.
> Below is the stack trace:
> {code}
> 14/02/27 08:32:35 ERROR storage.BlockManagerWorker: Exception handling buffer
> message
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
> at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:89)
> at
> org.apache.spark.storage.BlockManager.getLocalBytes(BlockManager.scala:285)
> at
> org.apache.spark.storage.BlockManagerWorker.getBlock(BlockManagerWorker.scala:90)
> at
> org.apache.spark.storage.BlockManagerWorker.processBlockMessage(BlockManagerWorker.scala:69)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$2.apply(BlockManagerWorker.scala:44)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at
> org.apache.spark.storage.BlockMessageArray.foreach(BlockMessageArray.scala:28)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at
> org.apache.spark.storage.BlockMessageArray.map(BlockMessageArray.scala:28)
> at
> org.apache.spark.storage.BlockManagerWorker.onBlockMessageReceive(BlockManagerWorker.scala:44)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34)
> at
> org.apache.spark.storage.BlockManagerWorker$$anonfun$1.apply(BlockManagerWorker.scala:34)
> at
> org.apache.spark.network.ConnectionManager.org$apache$spark$network$ConnectionManager$$handleMessage(ConnectionManager.scala:512)
> at
> org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:478)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> {code}
> And the JVM error log had a bunch of entries like this:
> {code}
> 7f4b48f89000-7f4b48f8a000 r--s ca:30 1622077901
> /mnt4/spark/spark-local-20140227020022-227c/26/shuffle_0_22312_38
> 7f4b48f8a000-7f4b48f8b000 r--s ca:20 545892715
> /mnt3/spark/spark-local-20140227020022-5ef5/3a/shuffle_0_26808_20
>
[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983853#comment-15983853 ] Ismael Juma commented on SPARK-18057: - It's worth noting that no-one is working on that ticket at the moment, so a fix may take some time. And even if it lands soon, it's likely to be in 0.11.0.0 first (0.10.2.1 is being voted and will be out very soon). > Update structured streaming kafka from 10.0.1 to 10.2.0 > --- > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982683#comment-15982683 ] Ismael Juma commented on SPARK-18057: - Thanks for the clarification [~zsxwing], that's helpful. > Update structured streaming kafka from 10.0.1 to 10.2.0 > --- > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982153#comment-15982153 ] Ismael Juma commented on SPARK-18057: - [~helena], about KAFKA-4879, are you suggesting that it's a regression in 0.10.2? The behaviour described (blocking to ensure offsets for all partitions are retrieved) has been there since the new Java consumer was introduced in 0.9.0. > Update structured streaming kafka from 10.0.1 to 10.2.0 > --- > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982077#comment-15982077 ] Ismael Juma edited comment on SPARK-18057 at 4/24/17 11:07 PM: --- Hi. A few clarifications below. "Based on previous kafka client upgrades I wouldn't expect them to be binary compatible, so it's likely to cause someone problems if they were also making use of kafka client libraries in their spark job. Still may be the path of least resistance." We do strive for binary compatibility for APIs that are not marked as Unstable. The Java consumer was introduced in 0.9.0.0 and the APIs were marked as Unstable. There were incompatible changes between 0.9.0.0 and 0.10.0.0 (KIP-45), which is what the above comment is probably referring to, but no other incompatible changes after that. "For what it's worth, or not, I ran into a wire protocol incompatibility between 0.10.0.1 and 0.10.1.1 today. I suspect we'd find the same vs 0.10.2.0. It surprised me." As Michael has said, 0.10.2 clients are the first version that supports older brokers (0.10.0 and higher). Before 0.10.2, clients supported newer brokers (e.g. 0.8.2 clients support 0.10.2 brokers), but not older brokers. I hope that helps. was (Author: ijuma): Hi. A few clarifications below. "Based on previous kafka client upgrades I wouldn't expect them to be binary compatible, so it's likely to cause someone problems if they were also making use of kafka client libraries in their spark job. Still may be the path of least resistance." We do strive for binary compatibility for APIs that are not marked as Unstable. The Java consumer was introduced in 0.9.0.0 and the APIs were marked as Unstable. There were incompatible changes between 0.9.0.0 and 0.10.0.0 (KIP-45), which is what the above comment is probably referring to, but no other incompatible changes after that. "For what it's worth, or not, I ran into a wire protocol incompatibility between 0.10.0.1 and 0.10.1.1 today. I suspect we'd find the same vs 0.10.2.0. It surprised me." As Michael has said, 0.10.2 clients are the first version that supports older brokers (0.10.0 and higher). Before 0.10.2, clients supported newer brokers (e.g. 0.8.2 clients support 0.10.2), but not older brokers. I hope that helps. > Update structured streaming kafka from 10.0.1 to 10.2.0 > --- > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982077#comment-15982077 ] Ismael Juma commented on SPARK-18057: - Hi. A few clarifications below. "Based on previous kafka client upgrades I wouldn't expect them to be binary compatible, so it's likely to cause someone problems if they were also making use of kafka client libraries in their spark job. Still may be the path of least resistance." We do strive for binary compatibility for APIs that are not marked as Unstable. The Java consumer was introduced in 0.9.0.0 and the APIs were marked as Unstable. There were incompatible changes between 0.9.0.0 and 0.10.0.0 (KIP-45), which is what the above comment is probably referring to, but no other incompatible changes after that. "For what it's worth, or not, I ran into a wire protocol incompatibility between 0.10.0.1 and 0.10.1.1 today. I suspect we'd find the same vs 0.10.2.0. It surprised me." As Michael has said, 0.10.2 clients are the first version that supports older brokers (0.10.0 and higher). Before 0.10.2, clients supported newer brokers (e.g. 0.8.2 clients support 0.10.2), but not older brokers. I hope that helps. > Update structured streaming kafka from 10.0.1 to 10.2.0 > --- > > Key: SPARK-18057 > URL: https://issues.apache.org/jira/browse/SPARK-18057 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming >Reporter: Cody Koeninger > > There are a couple of relevant KIPs here, > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API
[
https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329044#comment-15329044
]
Ismael Juma commented on SPARK-12177:
-
{quote}
b) the real benefit is security - I am personally a little more biased towards
authentication (Kerberos) than encryption, so I was just waiting for delegation
tokens to land.
{quote}
It's worth mentioning that authentication is also supported via TLS. I am aware
of a number of people who are using TLS for both authentication and encryption.
So, the security benefit is available now for some people, at least.
> Update KafkaDStreams to new Kafka 0.10 Consumer API
> ---
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.6.0
>Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not
> compatible with old one. So, I added new consumer api. I made separate
> classes in package org.apache.spark.streaming.kafka.v09 with changed API. I
> didn't remove old classes for more backward compatibility. User will not need
> to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API
[
https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324126#comment-15324126
]
Ismael Juma commented on SPARK-12177:
-
{quote}
I agree with Cody that the new Kafka consumer API implementation in Spark,
doesn't really have a benefit right now since we can't use the security
features which are gated by delegation tokens support in Kafka (KAFKA-1696).
{quote}
[~mgrover], it seems to me that people who want to use TLS would benefit from
the new consumer right now and wouldn't have to wait for delegation tokens.
What am I missing?
> Update KafkaDStreams to new Kafka 0.10 Consumer API
> ---
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
>Affects Versions: 1.6.0
>Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not
> compatible with old one. So, I added new consumer api. I made separate
> classes in package org.apache.spark.streaming.kafka.v09 with changed API. I
> didn't remove old classes for more backward compatibility. User will not need
> to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6363) make scala 2.11 default language
[ https://issues.apache.org/jira/browse/SPARK-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052631#comment-15052631 ] Ismael Juma commented on SPARK-6363: It's also worth pointing out that Scala 2.10 is no longer maintained since March 2015. > make scala 2.11 default language > > > Key: SPARK-6363 > URL: https://issues.apache.org/jira/browse/SPARK-6363 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: antonkulaga >Priority: Minor > Labels: scala > > Mostly all libraries already moved to 2.11 and many are starting to drop 2.10 > support. So, it will be better if Spark binaries would be build with Scala > 2.11 by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-558) Simplify run script by relying on sbt to launch app
[ https://issues.apache.org/jira/browse/SPARK-558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129801#comment-14129801 ] Ismael Juma commented on SPARK-558: --- Probably stale, yes. > Simplify run script by relying on sbt to launch app > --- > > Key: SPARK-558 > URL: https://issues.apache.org/jira/browse/SPARK-558 > Project: Spark > Issue Type: Improvement >Reporter: Ismael Juma > > The run script replicates SBT's functionality in order to build the classpath. > This could be avoided by creating a task in sbt that is responsible for > calling the appropriate main method, configuring the environment variables > from the script and then invoking sbt with the task name and arguments. > Is there a reason why we should not do this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
