[jira] [Updated] (SPARK-21219) Task retry occurs on same executor due to race condition with blacklisting

2017-07-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21219: Fix Version/s: 2.2.1 > Task retry occurs on same executor due to race condition with blacklisting >

[jira] [Assigned] (SPARK-19270) Add summary table to GLM summary

2017-07-11 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-19270: --- Assignee: Wayne Zhang > Add summary table to GLM summary >

[jira] [Commented] (SPARK-21362) Add JDBCDialect for Apache Drill

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083506#comment-16083506 ] Apache Spark commented on SPARK-21362: -- User 'radford1' has created a pull request f

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Everett Anderson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083429#comment-16083429 ] Everett Anderson commented on SPARK-21380: -- [~dongjoon] Hey -- I don't totally f

[jira] [Assigned] (SPARK-21362) Add JDBCDialect for Apache Drill

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21362: Assignee: Apache Spark > Add JDBCDialect for Apache Drill > --

[jira] [Assigned] (SPARK-21362) Add JDBCDialect for Apache Drill

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21362: Assignee: (was: Apache Spark) > Add JDBCDialect for Apache Drill > ---

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083434#comment-16083434 ] Dongjoon Hyun commented on SPARK-21380: --- One row in real normal table is okay. Your

[jira] [Comment Edited] (SPARK-20641) Key-value store abstraction and implementation for storing application data

2017-07-11 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083452#comment-16083452 ] Reynold Xin edited comment on SPARK-20641 at 7/12/17 5:06 AM: -

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2017-07-11 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083453#comment-16083453 ] Reynold Xin commented on SPARK-18085: - This is just large enough to warrant / deserve

[jira] [Comment Edited] (SPARK-20641) Key-value store abstraction and implementation for storing application data

2017-07-11 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083452#comment-16083452 ] Reynold Xin edited comment on SPARK-20641 at 7/12/17 5:05 AM: -

[jira] [Commented] (SPARK-20641) Key-value store abstraction and implementation for storing application data

2017-07-11 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083452#comment-16083452 ] Reynold Xin commented on SPARK-20641: - BTW why are we not using RocksDB? I saw that y

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Everett Anderson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083439#comment-16083439 ] Everett Anderson commented on SPARK-21380: -- Ah, I see. Okay, that makes sense. T

[jira] [Comment Edited] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083434#comment-16083434 ] Dongjoon Hyun edited comment on SPARK-21380 at 7/12/17 4:39 AM: ---

[jira] [Created] (SPARK-21385) hive-thriftserver register too many listener in listenerbus

2017-07-11 Thread honestman (JIRA)
honestman created SPARK-21385: - Summary: hive-thriftserver register too many listener in listenerbus Key: SPARK-21385 URL: https://issues.apache.org/jira/browse/SPARK-21385 Project: Spark Issue

[jira] [Updated] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-11 Thread Arun Achuthan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Achuthan updated SPARK-21303: -- Attachment: Executors-2017-07-11 at 6.44.12 PM.png Persist Incoming Event Stream

[jira] [Created] (SPARK-21384) Spark 2.2 + YARN without spark.yarn.jars / spark.yarn.archive fails

2017-07-11 Thread holdenk (JIRA)
holdenk created SPARK-21384: --- Summary: Spark 2.2 + YARN without spark.yarn.jars / spark.yarn.archive fails Key: SPARK-21384 URL: https://issues.apache.org/jira/browse/SPARK-21384 Project: Spark Is

[jira] [Updated] (SPARK-21374) Reading globbed paths from S3 into DF doesn't work if filesystem caching is disabled

2017-07-11 Thread Andrey Taptunov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Taptunov updated SPARK-21374: Description: *Motivation:* Filesystem configuration is not part of cache's key which is use

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Affects Version/s: (was: 2.1.0) 2.2.0 > Jars pulled from "--packages" ar

[jira] [Commented] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-11 Thread Arun Achuthan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083360#comment-16083360 ] Arun Achuthan commented on SPARK-21303: --- Hi Guys, Thank you very much for sharin

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

2017-07-11 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083345#comment-16083345 ] Leif Walsh commented on SPARK-13534: See SPARK-21190 for a case we're considering for

[jira] [Comment Edited] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves edited comment on SPARK-21383 at 7/12/17 2:17 AM: ---

[jira] [Assigned] (SPARK-21382) The note about Scala 2.10 in building-spark.md is wrong.

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21382: Assignee: Apache Spark > The note about Scala 2.10 in building-spark.md is wrong. > -

[jira] [Commented] (SPARK-21382) The note about Scala 2.10 in building-spark.md is wrong.

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083326#comment-16083326 ] Apache Spark commented on SPARK-21382: -- User 'liu-zhaokun' has created a pull reques

[jira] [Assigned] (SPARK-21382) The note about Scala 2.10 in building-spark.md is wrong.

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21382: Assignee: (was: Apache Spark) > The note about Scala 2.10 in building-spark.md is wro

[jira] [Commented] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves commented on SPARK-21383: --- Note we saw this with dynamic allocation o

[jira] [Updated] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21383: -- Summary: YARN can allocate to many executors (was: YARN: can allocate to many containers) > Y

[jira] [Created] (SPARK-21383) YARN: can allocate to many containers

2017-07-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-21383: - Summary: YARN: can allocate to many containers Key: SPARK-21383 URL: https://issues.apache.org/jira/browse/SPARK-21383 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-21382) The note about Scala 2.10 in building-spark.md is wrong.

2017-07-11 Thread liuzhaokun (JIRA)
liuzhaokun created SPARK-21382: -- Summary: The note about Scala 2.10 in building-spark.md is wrong. Key: SPARK-21382 URL: https://issues.apache.org/jira/browse/SPARK-21382 Project: Spark Issue T

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083272#comment-16083272 ] Dongjoon Hyun commented on SPARK-21380: --- Your case are too simple, so it's optimize

[jira] [Assigned] (SPARK-21221) CrossValidator and TrainValidationSplit Persist Nested Estimators

2017-07-11 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-21221: - Assignee: Ajay Saini > CrossValidator and TrainValidationSplit Persist Nested Es

[jira] [Created] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Suresh Purusothaman (JIRA)
Suresh Purusothaman created SPARK-21379: --- Summary: skip.header.line.count is ignored in HiveContext Key: SPARK-21379 URL: https://issues.apache.org/jira/browse/SPARK-21379 Project: Spark

[jira] [Assigned] (SPARK-21381) SparkR: pass on setHandleInvalid for classification algorithms

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21381: Assignee: (was: Apache Spark) > SparkR: pass on setHandleInvalid for classification al

[jira] [Closed] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21380. - Resolution: Not A Problem I'm closing this issue for now since the pattern works correct. If you

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083266#comment-16083266 ] Dongjoon Hyun commented on SPARK-21380: --- Hi, [~everett]. It's the correct result of

[jira] [Commented] (SPARK-21381) SparkR: pass on setHandleInvalid for classification algorithms

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083258#comment-16083258 ] Apache Spark commented on SPARK-21381: -- User 'wangmiao1981' has created a pull reque

[jira] [Created] (SPARK-21381) SparkR: pass on setHandleInvalid for classification algorithms

2017-07-11 Thread Miao Wang (JIRA)
Miao Wang created SPARK-21381: - Summary: SparkR: pass on setHandleInvalid for classification algorithms Key: SPARK-21381 URL: https://issues.apache.org/jira/browse/SPARK-21381 Project: Spark Iss

[jira] [Assigned] (SPARK-21381) SparkR: pass on setHandleInvalid for classification algorithms

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21381: Assignee: Apache Spark > SparkR: pass on setHandleInvalid for classification algorithms >

[jira] [Resolved] (SPARK-19285) Java - Provide user-defined function of 0 arguments (UDF0)

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19285. - Resolution: Fixed Fix Version/s: 2.3.0 > Java - Provide user-defined function of 0 arguments (UDF0

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Everett Anderson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083233#comment-16083233 ] Everett Anderson commented on SPARK-21380: -- [~dongjoon] Sure thing! I'll update

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083230#comment-16083230 ] Dongjoon Hyun commented on SPARK-21380: --- Hi, [~everett]. Thank you for reporting. T

[jira] [Updated] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21379: -- Component/s: (was: Spark Core) SQL > skip.header.line.count is ignored in

[jira] [Commented] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083186#comment-16083186 ] Dongjoon Hyun commented on SPARK-21379: --- This is the PR and the discussion in the c

[jira] [Closed] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-21379. - Resolution: Won't Fix Hi, [~spurusotha...@uk.imshealth.com]. This is a duplication of the previou

[jira] [Created] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-11 Thread Everett Anderson (JIRA)
Everett Anderson created SPARK-21380: Summary: Join with Columns thinks inner join is cross join even when aliased Key: SPARK-21380 URL: https://issues.apache.org/jira/browse/SPARK-21380 Project:

[jira] [Updated] (SPARK-21379) skip.header.line.count is ignored in HiveContext

2017-07-11 Thread Suresh Purusothaman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Purusothaman updated SPARK-21379: Description: We have an issue with Spark Job Server and Hive Context that it ignore

[jira] [Commented] (SPARK-21219) Task retry occurs on same executor due to race condition with blacklisting

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083134#comment-16083134 ] Apache Spark commented on SPARK-21219: -- User 'jsoltren' has created a pull request f

[jira] [Resolved] (SPARK-14663) Parse escape sequences in spark-defaults.conf

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-14663. Resolution: Not A Problem Those are Java property files, and you can use unicode escapes fo

[jira] [Assigned] (SPARK-18598) Encoding a Java Bean with extra accessors, produces inconsistent Dataset, resulting in AssertionError

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-18598: --- Assignee: (was: Xiao Li) > Encoding a Java Bean with extra accessors, produces inconsistent Data

[jira] [Resolved] (SPARK-18598) Encoding a Java Bean with extra accessors, produces inconsistent Dataset, resulting in AssertionError

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-18598. - Resolution: Fixed Fix Version/s: 2.3.0 > Encoding a Java Bean with extra accessors, produces incon

[jira] [Updated] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-11 Thread Ambud Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ambud Sharma updated SPARK-21378: - Description: Kafka direct stream fails with poll timeout: {code:java} JavaInputDStream> stream =

[jira] [Updated] (SPARK-19285) Java - Provide user-defined function of 0 arguments (UDF0)

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19285: Priority: Major (was: Minor) > Java - Provide user-defined function of 0 arguments (UDF0) > --

[jira] [Closed] (SPARK-18598) Encoding a Java Bean with extra accessors, produces inconsistent Dataset, resulting in AssertionError

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-18598. --- Resolution: Unresolved Fix Version/s: (was: 2.3.0) > Encoding a Java Bean with extra accessors, pr

[jira] [Reopened] (SPARK-18598) Encoding a Java Bean with extra accessors, produces inconsistent Dataset, resulting in AssertionError

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-18598: - > Encoding a Java Bean with extra accessors, produces inconsistent Dataset, > resulting in AssertionError >

[jira] [Assigned] (SPARK-18598) Encoding a Java Bean with extra accessors, produces inconsistent Dataset, resulting in AssertionError

2017-07-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-18598: --- Assignee: Xiao Li > Encoding a Java Bean with extra accessors, produces inconsistent Dataset, > res

[jira] [Updated] (SPARK-20682) Support a new faster ORC data source based on Apache ORC

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20682: -- Affects Version/s: 2.2.0 > Support a new faster ORC data source based on Apache ORC > -

[jira] [Updated] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-11 Thread Ambud Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ambud Sharma updated SPARK-21378: - Description: Kafka direct stream fails with poll timeout: {code:java} JavaInputDStream> stream =

[jira] [Commented] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083102#comment-16083102 ] Apache Spark commented on SPARK-21370: -- User 'brkyvz' has created a pull request for

[jira] [Updated] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-11 Thread Ambud Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ambud Sharma updated SPARK-21378: - Description: Kafka direct stream fails with poll timeout: {code:java} JavaInputDStream> stream =

[jira] [Created] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-11 Thread Ambud Sharma (JIRA)
Ambud Sharma created SPARK-21378: Summary: Spark Poll timeout when specific offsets are passed Key: SPARK-21378 URL: https://issues.apache.org/jira/browse/SPARK-21378 Project: Spark Issue Typ

[jira] [Updated] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21370: Description: During Streaming Aggregation, we have two StateStores per task, one used as read-only

[jira] [Updated] (SPARK-20901) Feature parity for ORC with Parquet

2017-07-11 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20901: -- Affects Version/s: 2.2.0 > Feature parity for ORC with Parquet > --

[jira] [Resolved] (SPARK-10610) Using AppName instead of AppId in the name of all metrics

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-10610. Resolution: Duplicate Pretty sure you can do that now by setting "spark.metrics.namespace".

[jira] [Reopened] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reopened SPARK-21370: - > Avoid doing anything on HDFSBackedStateStore.abort() when there are no > updates to commit > -

[jira] [Updated] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21370: Issue Type: Test (was: Improvement) > Avoid doing anything on HDFSBackedStateStore.abort() when th

[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083081#comment-16083081 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 10:13 PM:

[jira] [Resolved] (SPARK-7108) spark.local.dir is no longer honored in Standalone mode

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-7108. --- Resolution: Not A Problem I'm going to close this because I don't see this as an issue, and I

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083085#comment-16083085 ] Marcelo Vanzin commented on SPARK-21377: bq. So your suggestion is that we use an

[jira] [Commented] (SPARK-21219) Task retry occurs on same executor due to race condition with blacklisting

2017-07-11 Thread Jose Soltren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083082#comment-16083082 ] Jose Soltren commented on SPARK-21219: -- I think it would be good to backport this to

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083081#comment-16083081 ] Saisai Shao commented on SPARK-21377: - My original purpose is to add jars uploaded by

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083060#comment-16083060 ] Saisai Shao commented on SPARK-21377: - Thanks [~vanzin] for your comment. Your comme

[jira] [Resolved] (SPARK-6355) Spark standalone cluster does not support local:/ url for jar file

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-6355. --- Resolution: Fixed I'm going to trust the last comment about this being fixed. > Spark standal

[jira] [Comment Edited] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-11 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082780#comment-16082780 ] shane knapp edited comment on SPARK-21367 at 7/11/17 9:29 PM: -

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083063#comment-16083063 ] Marcelo Vanzin commented on SPARK-21377: bq. my original thought is to add main j

[jira] [Comment Edited] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083060#comment-16083060 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 9:54 PM: -

[jira] [Comment Edited] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-11 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082757#comment-16082757 ] shane knapp edited comment on SPARK-21367 at 7/11/17 9:34 PM: -

[jira] [Comment Edited] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-11 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082757#comment-16082757 ] shane knapp edited comment on SPARK-21367 at 7/11/17 9:34 PM: -

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083033#comment-16083033 ] Marcelo Vanzin commented on SPARK-21377: bq. we specify HBase related jars with -

[jira] [Comment Edited] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-07-11 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082757#comment-16082757 ] shane knapp edited comment on SPARK-21367 at 7/11/17 9:34 PM: -

[jira] [Commented] (SPARK-21190) SPIP: Vectorized UDFs in Python

2017-07-11 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083018#comment-16083018 ] Bryan Cutler commented on SPARK-21190: -- [~cloud_fan] yes, I know not every function

[jira] [Assigned] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21377: Assignee: Apache Spark > Add a new configuration to extend AM classpath in yarn client mod

[jira] [Updated] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Description: In this issue we have a long running Spark application with secure HBase, which requi

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082990#comment-16082990 ] Apache Spark commented on SPARK-21377: -- User 'jerryshao' has created a pull request

[jira] [Assigned] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21377: Assignee: (was: Apache Spark) > Add a new configuration to extend AM classpath in yarn

[jira] [Updated] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Summary: Add a new configuration to extend AM classpath in yarn client mode (was: Jars pulled from

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Description: STR: * Set below config in spark-default.conf {code} spark.yarn.security.credentials.h

[jira] [Comment Edited] (SPARK-21067) Thrift Server - CTAS fail with Unable to move source

2017-07-11 Thread Dominic Ricard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080503#comment-16080503 ] Dominic Ricard edited comment on SPARK-21067 at 7/11/17 8:00 PM: --

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

2017-07-11 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082900#comment-16082900 ] Ruslan Dautkhanov commented on SPARK-13534: --- So Apache Arrow would currently be

[jira] [Issue Comment Deleted] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2017-07-11 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-21274: -- Comment: was deleted (was: [~rxin], I wish I could. We only use PySpark and SQL API to

[jira] [Commented] (SPARK-20263) create empty dataframes in sparkR

2017-07-11 Thread Ott Toomet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082882#comment-16082882 ] Ott Toomet commented on SPARK-20263: Grishma--sure, there are workarounds like that.

[jira] [Updated] (SPARK-21221) CrossValidator and TrainValidationSplit Persist Nested Estimators

2017-07-11 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-21221: -- Affects Version/s: (was: 2.1.1) 2.2.0 > CrossValidator and T

[jira] [Commented] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082864#comment-16082864 ] Saisai Shao commented on SPARK-21377: - [~srowen] this is a separate issue to SPARK-21

[jira] [Commented] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082873#comment-16082873 ] Saisai Shao commented on SPARK-21377: - SPARK-21376 and here are both security issues,

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Priority: Minor (was: Major) > Jars pulled from "--packages" are not added into AM classpath > ---

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Priority: Minor (was: Major) > Token is not renewed in yarn client process in cluster mode > -

[jira] [Updated] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Component/s: (was: Spark Core) YARN > Jars pulled from "--packages" are not ad

[jira] [Comment Edited] (SPARK-21377) Jars pulled from "--packages" are not added into AM classpath

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082864#comment-16082864 ] Saisai Shao edited comment on SPARK-21377 at 7/11/17 7:58 PM: -

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Affects Version/s: (was: 2.1.0) 2.2.0 2.1.1 > Tok

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082857#comment-16082857 ] Saisai Shao commented on SPARK-21376: - I will work on this, thanks [~yeshavora]. > T

[jira] [Updated] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21376: Component/s: (was: Spark Core) YARN > Token is not renewed in yarn client proc

[jira] [Updated] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-11 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21375: - Description: Date and timestamp are not yet supported in DataFrame.toPandas() using ArrowConvert

  1   2   >