[jira] [Updated] (SPARK-31584) NullPointerException when parsing event log with InMemoryStore
[ https://issues.apache.org/jira/browse/SPARK-31584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31584: Fix Version/s: 3.1.0 > NullPointerException when parsing event log with InMemoryStore > -- > > Key: SPARK-31584 > URL: https://issues.apache.org/jira/browse/SPARK-31584 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.0.1 >Reporter: Baohe Zhang >Assignee: Baohe Zhang >Priority: Minor > Fix For: 3.0.1, 3.1.0 > > Attachments: errorstack.txt > > > I compiled with the current branch-3.0 source and tested it in mac os. A > java.lang.NullPointerException will be thrown when below conditions are met: > # Using InMemoryStore as kvstore when parsing the event log file (e.g., when > spark.history.store.path is unset). > # At least one stage in this event log has task number greater than > spark.ui.retainedTasks (by default is 10). In this case, kvstore needs to > delete extra task records. > # The job has more than one stage, so parentToChildrenMap in > InMemoryStore.java will have more than one key. > The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In > the method deleteParentIndex(). > {code:java} > private void deleteParentIndex(Object key) { > if (hasNaturalParentIndex) { > for (NaturalKeys v : parentToChildrenMap.values()) { > if (v.remove(asKey(key))) { > // `v` can be empty after removing the natural key and we can > remove it from > // `parentToChildrenMap`. However, `parentToChildrenMap` is a > ConcurrentMap and such > // checking and deleting can be slow. > // This method is to delete one object with certain key, let's > make it simple here. > break; > } > } > } > }{code} > In “if (v.remove(asKey(key)))”, if the key is not contained in v, > "v.remove(asKey(key))" will return null, and java will throw a > NullPointerException when executing "if (null)". > An exception stack trace is attached. > This issue can be fixed by updating if statement to > {code:java} > if (v.remove(asKey(key)) != null){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31584) NullPointerException when parsing event log with InMemoryStore
[ https://issues.apache.org/jira/browse/SPARK-31584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Baohe Zhang updated SPARK-31584: Attachment: errorstack.txt > NullPointerException when parsing event log with InMemoryStore > -- > > Key: SPARK-31584 > URL: https://issues.apache.org/jira/browse/SPARK-31584 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 3.0.1 >Reporter: Baohe Zhang >Priority: Minor > Fix For: 3.0.1 > > Attachments: errorstack.txt > > > I compiled with the current branch-3.0 source and tested it in mac os. A > java.lang.NullPointerException will be thrown when below conditions are met: > # Using InMemoryStore as kvstore when parsing the event log file (e.g., when > spark.history.store.path is unset). > # At least one stage in this event log has task number greater than > spark.ui.retainedTasks (by default is 10). In this case, kvstore needs to > delete extra task records. > # The job has more than one stage, so parentToChildrenMap in > InMemoryStore.java will have more than one key. > The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In > the method deleteParentIndex(). > {code:java} > private void deleteParentIndex(Object key) { > if (hasNaturalParentIndex) { > for (NaturalKeys v : parentToChildrenMap.values()) { > if (v.remove(asKey(key))) { > // `v` can be empty after removing the natural key and we can > remove it from > // `parentToChildrenMap`. However, `parentToChildrenMap` is a > ConcurrentMap and such > // checking and deleting can be slow. > // This method is to delete one object with certain key, let's > make it simple here. > break; > } > } > } > }{code} > In “if (v.remove(asKey(key)))”, if the key is not contained in v, > "v.remove(asKey(key))" will return null, and java will throw a > NullPointerException when executing "if (null)". > An exception stack trace is attached. > This issue can be fixed by updating if statement to > {code:java} > if (v.remove(asKey(key)) != null){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org