[jira] [Created] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo cause

Ricky Yang created SPARK-12601: -- Summary: worker output a large number of log when size RollingPolicy shouldRollover method use loginfo cause Key: SPARK-12601 URL:

[jira] [Commented] (SPARK-10963) Make KafkaCluster api public

[ https://issues.apache.org/jira/browse/SPARK-10963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076464#comment-15076464 ] Youcef HILEM commented on SPARK-10963: -- I doubt that this class will be made public. The recent

[jira] [Created] (SPARK-12599) Remove the use of the deprecated callUDF in MLlib

Reynold Xin created SPARK-12599: --- Summary: Remove the use of the deprecated callUDF in MLlib Key: SPARK-12599 URL: https://issues.apache.org/jira/browse/SPARK-12599 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo

[ https://issues.apache.org/jira/browse/SPARK-12601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076412#comment-15076412 ] Ricky Yang commented on SPARK-12601: /** Should rollover if the next set of bytes is going to

[jira] [Updated] (SPARK-12594) Outer Join Elimination by Local Predicates

[ https://issues.apache.org/jira/browse/SPARK-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12594: Summary: Outer Join Elimination by Local Predicates (was: Outer Join Elimination: Outer to

[jira] [Resolved] (SPARK-12592) TestHive.reset hides Spark testing logs

[ https://issues.apache.org/jira/browse/SPARK-12592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12592. - Resolution: Fixed Assignee: Cheng Lian Fix Version/s: 2.0.0 > TestHive.reset

[jira] [Resolved] (SPARK-12284) Output UnsafeRow from window function

[ https://issues.apache.org/jira/browse/SPARK-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12284. - Resolution: Fixed Assignee: Davies Liu Fix Version/s: 2.0.0 > Output UnsafeRow

[jira] [Resolved] (SPARK-12286) Support UnsafeRow in all SparkPlan (if possible)

[ https://issues.apache.org/jira/browse/SPARK-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12286. - Resolution: Fixed Fix Version/s: 2.0.0 > Support UnsafeRow in all SparkPlan (if possible)

[jira] [Resolved] (SPARK-12290) Change the default value in SparkPlan

[ https://issues.apache.org/jira/browse/SPARK-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12290. - Resolution: Fixed Assignee: Davies Liu Fix Version/s: 2.0.0 > Change the default

[jira] [Resolved] (SPARK-12294) Support UnsafeRow in HiveTableScan

[ https://issues.apache.org/jira/browse/SPARK-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12294. - Resolution: Fixed Assignee: Davies Liu Fix Version/s: 2.0.0 > Support UnsafeRow

[jira] [Commented] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo

[ https://issues.apache.org/jira/browse/SPARK-12601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076432#comment-15076432 ] Ricky Yang commented on SPARK-12601: https://github.com/apache/spark/pull/10549/files > worker

[jira] [Updated] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

[ https://issues.apache.org/jira/browse/SPARK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12602: Description: If applicable, we can push Inner Join through Outer Join. This can reduce the number of rows

[jira] [Created] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

Xiao Li created SPARK-12602: --- Summary: Join Reordering: Pushing Inner Join Through Outer Join Key: SPARK-12602 URL: https://issues.apache.org/jira/browse/SPARK-12602 Project: Spark Issue Type:

[jira] [Updated] (SPARK-12594) Outer Join Elimination: Outer to Inner/Left/Right, Right to Inner and Left to Inner

[ https://issues.apache.org/jira/browse/SPARK-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12594: Description: Elimination of outer joins, if the local predicates can restrict the result sets so that all

[jira] [Updated] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

[ https://issues.apache.org/jira/browse/SPARK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12602: Description: If applicable, we can push Inner Join through Outer Join. The reordering can reduce the

[jira] [Updated] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

[ https://issues.apache.org/jira/browse/SPARK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-12602: Description: If applicable, we can push Inner Join through Outer Join. This can reduce the number of rows

[jira] [Resolved] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo

[ https://issues.apache.org/jira/browse/SPARK-12601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricky Yang resolved SPARK-12601. Resolution: Fixed Fix Version/s: 1.6.1 1.6.0 /** Should rollover if the

[jira] [Updated] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo

[ https://issues.apache.org/jira/browse/SPARK-12601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricky Yang updated SPARK-12601: --- Description: when using size RollingPolicythis code cause worker output a large of log .it shoud be

[jira] [Updated] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo

[ https://issues.apache.org/jira/browse/SPARK-12601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricky Yang updated SPARK-12601: --- Description: worker output a large number of log when size RollingPolicy shouldRollover was:

[jira] [Assigned] (SPARK-12599) Remove the use of the deprecated callUDF in MLlib

[ https://issues.apache.org/jira/browse/SPARK-12599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12599: Assignee: Apache Spark (was: Reynold Xin) > Remove the use of the deprecated callUDF in

[jira] [Created] (SPARK-12600) Remove deprecated methods in SQL / DataFrames

Reynold Xin created SPARK-12600: --- Summary: Remove deprecated methods in SQL / DataFrames Key: SPARK-12600 URL: https://issues.apache.org/jira/browse/SPARK-12600 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12599) Remove the use of the deprecated callUDF in MLlib

[ https://issues.apache.org/jira/browse/SPARK-12599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076380#comment-15076380 ] Apache Spark commented on SPARK-12599: -- User 'rxin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12599) Remove the use of the deprecated callUDF in MLlib

[ https://issues.apache.org/jira/browse/SPARK-12599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12599: Assignee: Reynold Xin (was: Apache Spark) > Remove the use of the deprecated callUDF in

[jira] [Assigned] (SPARK-12600) Remove deprecated methods in SQL / DataFrames

[ https://issues.apache.org/jira/browse/SPARK-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12600: Assignee: Apache Spark (was: Reynold Xin) > Remove deprecated methods in SQL /

[jira] [Assigned] (SPARK-12600) Remove deprecated methods in SQL / DataFrames

[ https://issues.apache.org/jira/browse/SPARK-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12600: Assignee: Reynold Xin (was: Apache Spark) > Remove deprecated methods in SQL /

[jira] [Commented] (SPARK-12600) Remove deprecated methods in SQL / DataFrames

[ https://issues.apache.org/jira/browse/SPARK-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076385#comment-15076385 ] Apache Spark commented on SPARK-12600: -- User 'rxin' has created a pull request for this issue:

[jira] [Updated] (SPARK-12601) worker output a large number of log when size RollingPolicy shouldRollover method use loginfo

[ https://issues.apache.org/jira/browse/SPARK-12601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricky Yang updated SPARK-12601: --- Summary: worker output a large number of log when size RollingPolicy shouldRollover method use

[jira] [Assigned] (SPARK-12286) Support UnsafeRow in all SparkPlan (if possible)

[ https://issues.apache.org/jira/browse/SPARK-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12286: Assignee: Davies Liu (was: Apache Spark) > Support UnsafeRow in all SparkPlan (if

[jira] [Assigned] (SPARK-12286) Support UnsafeRow in all SparkPlan (if possible)

[ https://issues.apache.org/jira/browse/SPARK-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12286: Assignee: Apache Spark (was: Davies Liu) > Support UnsafeRow in all SparkPlan (if

[jira] [Reopened] (SPARK-12286) Support UnsafeRow in all SparkPlan (if possible)

[ https://issues.apache.org/jira/browse/SPARK-12286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reopened SPARK-12286: - Reopen due to test flakiness. > Support UnsafeRow in all SparkPlan (if possible) >

[jira] [Assigned] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

[ https://issues.apache.org/jira/browse/SPARK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12602: Assignee: Apache Spark > Join Reordering: Pushing Inner Join Through Outer Join >

[jira] [Commented] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

[ https://issues.apache.org/jira/browse/SPARK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076435#comment-15076435 ] Apache Spark commented on SPARK-12602: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-12602) Join Reordering: Pushing Inner Join Through Outer Join

[ https://issues.apache.org/jira/browse/SPARK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12602: Assignee: (was: Apache Spark) > Join Reordering: Pushing Inner Join Through Outer

[jira] [Created] (SPARK-12595) fold should pass arguments to op in the correct order

Maciej Szymkiewicz created SPARK-12595: -- Summary: fold should pass arguments to op in the correct order Key: SPARK-12595 URL: https://issues.apache.org/jira/browse/SPARK-12595 Project: Spark

[jira] [Created] (SPARK-12597) Use udf replace callUDF for ML

Yanbo Liang created SPARK-12597: --- Summary: Use udf replace callUDF for ML Key: SPARK-12597 URL: https://issues.apache.org/jira/browse/SPARK-12597 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-12597) Use udf replace callUDF for ML

[ https://issues.apache.org/jira/browse/SPARK-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12597: Description: callUDF has been deprecated and will be removed in Spark 2.0. We should replace the

[jira] [Updated] (SPARK-12597) Use udf to replace callUDF for ML

[ https://issues.apache.org/jira/browse/SPARK-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-12597: Summary: Use udf to replace callUDF for ML (was: Use udf replace callUDF for ML) > Use udf to

[jira] [Commented] (SPARK-11714) Make Spark on Mesos honor port restrictions

[ https://issues.apache.org/jira/browse/SPARK-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076309#comment-15076309 ] Stavros Kontopoulos commented on SPARK-11714: - Before moving on with a PR i was thinking of

[jira] [Resolved] (SPARK-12595) fold should pass arguments to op in the correct order

[ https://issues.apache.org/jira/browse/SPARK-12595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12595. --- Resolution: Duplicate > fold should pass arguments to op in the correct order >

[jira] [Commented] (SPARK-8555) Online Variational Inference for the Hierarchical Dirichlet Process

[ https://issues.apache.org/jira/browse/SPARK-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076303#comment-15076303 ] Sean Owen commented on SPARK-8555: -- I doubt this would be merged into Spark; at least, create the package

[jira] [Resolved] (SPARK-12596) spark.sql.autoBroadcastJoinThreshold should be int

[ https://issues.apache.org/jira/browse/SPARK-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12596. --- Resolution: Not A Problem The maximum int is 2^31-1 ~= 2.1GB, not 4.2GB. The behavior is correct. >

[jira] [Assigned] (SPARK-12597) Use udf to replace callUDF for ML

[ https://issues.apache.org/jira/browse/SPARK-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12597: Assignee: (was: Apache Spark) > Use udf to replace callUDF for ML >

[jira] [Assigned] (SPARK-12597) Use udf to replace callUDF for ML

[ https://issues.apache.org/jira/browse/SPARK-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12597: Assignee: Apache Spark > Use udf to replace callUDF for ML >

[jira] [Updated] (SPARK-7683) Confusing behavior of fold function of RDD in pyspark

[ https://issues.apache.org/jira/browse/SPARK-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7683: - Issue Type: Sub-task (was: Bug) Parent: SPARK-11806 > Confusing behavior of fold function of RDD

[jira] [Created] (SPARK-12596) spark.sql.autoBroadcastJoinThreshold should be int

Srinivasarao Daruna created SPARK-12596: --- Summary: spark.sql.autoBroadcastJoinThreshold should be int Key: SPARK-12596 URL: https://issues.apache.org/jira/browse/SPARK-12596 Project: Spark

[jira] [Commented] (SPARK-12597) Use udf to replace callUDF for ML

[ https://issues.apache.org/jira/browse/SPARK-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076310#comment-15076310 ] Apache Spark commented on SPARK-12597: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Commented] (SPARK-6416) RDD.fold() requires the operator to be commutative

[ https://issues.apache.org/jira/browse/SPARK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076318#comment-15076318 ] Sean Owen commented on SPARK-6416: -- [~joshrosen] [~markhamstra] WDYT about deprecating fold() for 2.x? or

[jira] [Updated] (SPARK-7683) Confusing behavior of fold function of RDD in pyspark

[ https://issues.apache.org/jira/browse/SPARK-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7683: - Labels: releasenotes (was: ) > Confusing behavior of fold function of RDD in pyspark >

[jira] [Commented] (SPARK-6416) RDD.fold() requires the operator to be commutative

[ https://issues.apache.org/jira/browse/SPARK-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076367#comment-15076367 ] Mark Hamstra commented on SPARK-6416: - I don't see any reason to change the API wrt `fold`. With

[jira] [Created] (SPARK-12598) Bug in setMinPartitions function of StreamFileInputFormat

Darek Blasiak created SPARK-12598: - Summary: Bug in setMinPartitions function of StreamFileInputFormat Key: SPARK-12598 URL: https://issues.apache.org/jira/browse/SPARK-12598 Project: Spark

[jira] [Assigned] (SPARK-12533) hiveContext.table() throws the wrong exception

[ https://issues.apache.org/jira/browse/SPARK-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12533: Assignee: Apache Spark > hiveContext.table() throws the wrong exception >

[jira] [Assigned] (SPARK-12533) hiveContext.table() throws the wrong exception

[ https://issues.apache.org/jira/browse/SPARK-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12533: Assignee: (was: Apache Spark) > hiveContext.table() throws the wrong exception >

[jira] [Commented] (SPARK-12533) hiveContext.table() throws the wrong exception

[ https://issues.apache.org/jira/browse/SPARK-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076331#comment-15076331 ] Apache Spark commented on SPARK-12533: -- User 'thomastechs' has created a pull request for this

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076344#comment-15076344 ] Apache Spark commented on SPARK-1537: - User 'steveloughran' has created a pull request for this issue:

[jira] [Commented] (SPARK-12533) hiveContext.table() throws the wrong exception

[ https://issues.apache.org/jira/browse/SPARK-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076327#comment-15076327 ] Jayadevan M commented on SPARK-12533: - Thomas, Updated my thoughts in the pull request. Pls check. >

[jira] [Comment Edited] (SPARK-12533) hiveContext.table() throws the wrong exception

[ https://issues.apache.org/jira/browse/SPARK-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076330#comment-15076330 ] Thomas Sebastian edited comment on SPARK-12533 at 1/1/16 4:43 PM: --

[jira] [Commented] (SPARK-12533) hiveContext.table() throws the wrong exception

[ https://issues.apache.org/jira/browse/SPARK-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15076330#comment-15076330 ] Thomas Sebastian commented on SPARK-12533: -- [~jayadevan.m], it is added. [~marmbrus] , The

[jira] [Resolved] (SPARK-12501) spark fails to build with mesos-0.26.0 (secret)

[ https://issues.apache.org/jira/browse/SPARK-12501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-12501. --- Resolution: Not A Problem > spark fails to build with mesos-0.26.0 (secret) >

[jira] [Resolved] (SPARK-10291) Add statsByKey method to compute StatCounters for each key in an RDD

[ https://issues.apache.org/jira/browse/SPARK-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-10291. --- Resolution: Won't Fix > Add statsByKey method to compute StatCounters for each key in an RDD >