[jira] [Commented] (SPARK-18823) Assignation by column name variable not available or bug?

2017-01-11 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820389#comment-15820389 ] Felix Cheung commented on SPARK-18823: -- Yap. I'll start on this shortly. > Assignation by column

[jira] [Commented] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820371#comment-15820371 ] Saisai Shao commented on SPARK-19179: - Thanks [~tgraves] to point out the left thing, let me handle

[jira] [Resolved] (SPARK-12076) countDistinct behaves inconsistently

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12076. -- Resolution: Cannot Reproduce I strongly think no one is going to reproduce this. I tried to

[jira] [Assigned] (SPARK-18693) BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator should use sample weight data

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18693: Assignee: Apache Spark > BinaryClassificationEvaluator, RegressionEvaluator, and >

[jira] [Assigned] (SPARK-18693) BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator should use sample weight data

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18693: Assignee: (was: Apache Spark) > BinaryClassificationEvaluator, RegressionEvaluator,

[jira] [Commented] (SPARK-18693) BinaryClassificationEvaluator, RegressionEvaluator, and MulticlassClassificationEvaluator should use sample weight data

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820273#comment-15820273 ] Apache Spark commented on SPARK-18693: -- User 'imatiach-msft' has created a pull request for this

[jira] [Resolved] (SPARK-16848) Check schema validation for user-specified schema in jdbc and table APIs

2017-01-11 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-16848. - Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.2.0 > Check schema validation

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 4:59 AM: - [~debasish83]

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 4:58 AM: - [~debasish83]

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820180#comment-15820180 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 4:54 AM: - As the detail

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820180#comment-15820180 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 4:54 AM: - As the detail

[jira] [Commented] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820180#comment-15820180 ] Weichen Xu commented on SPARK-10078: As the detail problems I list above(I only list a small part

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819851#comment-15819851 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 4:43 AM: - [~debasish83]

[jira] [Comment Edited] (SPARK-19051) test_hivecontext (pyspark.sql.tests.HiveSparkSubmitTests) fails in python/run-tests

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819957#comment-15819957 ] Hyukjin Kwon edited comment on SPARK-19051 at 1/12/17 4:36 AM: --- I just

[jira] [Updated] (SPARK-19188) Run spark in scala as script file, note not just REPL

2017-01-11 Thread wangzhihao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhihao updated SPARK-19188: --- Description: Hi, I'm looking for the feature to run spark/scala in script file. The current

[jira] [Updated] (SPARK-19188) Run spark in scala as script file, note not just REPL

2017-01-11 Thread wangzhihao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhihao updated SPARK-19188: --- Description: Hi, I'm looking for the feature to run spark/scala in script file. The current

[jira] [Created] (SPARK-19188) Run spark in scala as script file, note not just REPL

2017-01-11 Thread wangzhihao (JIRA)
wangzhihao created SPARK-19188: -- Summary: Run spark in scala as script file, note not just REPL Key: SPARK-19188 URL: https://issues.apache.org/jira/browse/SPARK-19188 Project: Spark Issue

[jira] [Updated] (SPARK-19133) SparkR glm Gamma family results in error

2017-01-11 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-19133: - Affects Version/s: 2.0.0 Target Version/s: 2.0.3, 2.1.1, 2.2.0 (was: 2.2.0) Fix

[jira] [Created] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist

2017-01-11 Thread roncenzhao (JIRA)
roncenzhao created SPARK-19187: -- Summary: querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist Key: SPARK-19187 URL:

[jira] [Created] (SPARK-19186) Hash symbol in middle of Sybase database table name causes Spark Exception

2017-01-11 Thread Adrian Schulewitz (JIRA)
Adrian Schulewitz created SPARK-19186: - Summary: Hash symbol in middle of Sybase database table name causes Spark Exception Key: SPARK-19186 URL: https://issues.apache.org/jira/browse/SPARK-19186

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 3:02 AM: - [~debasish83]

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 2:55 AM: - [~debasish83]

[jira] [Commented] (SPARK-14901) java exception when showing join

2017-01-11 Thread Brent Elmer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819984#comment-15819984 ] Brent Elmer commented on SPARK-14901: - Netezza is an IBM product so there is no place to download it

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 2:48 AM: - [~debasish83]

[jira] [Comment Edited] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu edited comment on SPARK-10078 at 1/12/17 2:45 AM: - [~debasish83]

[jira] [Commented] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819964#comment-15819964 ] Weichen Xu commented on SPARK-10078: [~debasish83] But when we implement VF-LBFGS/VF-OWLQN base on

[jira] [Commented] (SPARK-19051) test_hivecontext (pyspark.sql.tests.HiveSparkSubmitTests) fails in python/run-tests

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819957#comment-15819957 ] Hyukjin Kwon commented on SPARK-19051: -- I just if this JIRA says a flaky test or a constantly

[jira] [Resolved] (SPARK-17923) dateFormat unexpected kwarg to df.write.csv

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17923. -- Resolution: Duplicate This should be fixed in 2.0.1 and 2.1.0. > dateFormat unexpected kwarg

[jira] [Assigned] (SPARK-19184) Improve numerical stability for method tallSkinnyQR.

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19184: Assignee: Apache Spark > Improve numerical stability for method tallSkinnyQR. >

[jira] [Assigned] (SPARK-19184) Improve numerical stability for method tallSkinnyQR.

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19184: Assignee: (was: Apache Spark) > Improve numerical stability for method tallSkinnyQR.

[jira] [Commented] (SPARK-19184) Improve numerical stability for method tallSkinnyQR.

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819941#comment-15819941 ] Apache Spark commented on SPARK-19184: -- User 'hl475' has created a pull request for this issue:

[jira] [Commented] (SPARK-15407) Floor division

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819926#comment-15819926 ] Hyukjin Kwon commented on SPARK-15407: -- Hi [~hvanhovell], I just wonder if we should resolve this as

[jira] [Resolved] (SPARK-15251) Cannot apply PythonUDF to aggregated column

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-15251. -- Resolution: Cannot Reproduce {code} >>> def timesTwo(x): ... return x * 2 ... >>>

[jira] [Commented] (SPARK-14901) java exception when showing join

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819920#comment-15819920 ] Hyukjin Kwon commented on SPARK-14901: -- Would this be possible to provide a self-contained

[jira] [Created] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

2017-01-11 Thread Kalvin Chau (JIRA)
Kalvin Chau created SPARK-19185: --- Summary: ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing Key: SPARK-19185 URL: https://issues.apache.org/jira/browse/SPARK-19185 Project:

[jira] [Created] (SPARK-19184) Improve numerical stability for method tallSkinnyQR.

2017-01-11 Thread Huamin Li (JIRA)
Huamin Li created SPARK-19184: - Summary: Improve numerical stability for method tallSkinnyQR. Key: SPARK-19184 URL: https://issues.apache.org/jira/browse/SPARK-19184 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13303) Spark fails with pandas import error when pandas is not explicitly imported by user

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819894#comment-15819894 ] Hyukjin Kwon commented on SPARK-13303: -- +1 > Spark fails with pandas import error when pandas is

[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819883#comment-15819883 ] Hyukjin Kwon commented on SPARK-12717: -- It still happens in the current master. > pyspark broadcast

[jira] [Resolved] (SPARK-11428) Schema Merging Broken for Some Queries

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-11428. -- Resolution: Duplicate I am pretty sure that it is a duplicate of SPARK-11103. Please reopen

[jira] [Resolved] (SPARK-8128) Schema Merging Broken: Dataframe Fails to Recognize Column in Schema

2017-01-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8128. - Resolution: Duplicate I am pretty sure that it duplicates SPARK-11103. Please reopen this if

[jira] [Commented] (SPARK-10078) Vector-free L-BFGS

2017-01-11 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819851#comment-15819851 ] Weichen Xu commented on SPARK-10078: [~debasish83] Can L-BFGS-B be distributed computed when scaled

[jira] [Commented] (SPARK-19164) Review of UserDefinedFunction._broadcast

2017-01-11 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819820#comment-15819820 ] Reynold Xin commented on SPARK-19164: - Which one should I review? I see that you opened a bunch of

[jira] [Commented] (SPARK-19115) SparkSQL unsupports the command " create external table if not exist new_tbl like old_tbl"

2017-01-11 Thread Xiaochen Ouyang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819716#comment-15819716 ] Xiaochen Ouyang commented on SPARK-19115: - May I ask you whether Spark supports the following

[jira] [Assigned] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19180: Assignee: Apache Spark > the offset of short is 4 in OffHeapColumnVector's putShorts >

[jira] [Assigned] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19180: Assignee: (was: Apache Spark) > the offset of short is 4 in OffHeapColumnVector's

[jira] [Commented] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819634#comment-15819634 ] Apache Spark commented on SPARK-19180: -- User 'yucai' has created a pull request for this issue:

[jira] [Commented] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819623#comment-15819623 ] Apache Spark commented on SPARK-19183: -- User 'ericl' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19183: Assignee: (was: Apache Spark) > Add deleteWithJob hook to internal commit protocol

[jira] [Assigned] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19183: Assignee: Apache Spark > Add deleteWithJob hook to internal commit protocol API >

[jira] [Created] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API

2017-01-11 Thread Eric Liang (JIRA)
Eric Liang created SPARK-19183: -- Summary: Add deleteWithJob hook to internal commit protocol API Key: SPARK-19183 URL: https://issues.apache.org/jira/browse/SPARK-19183 Project: Spark Issue

[jira] [Updated] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

2017-01-11 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19182: - Summary: Optimize the lock in StreamingJobProgressListener to not block UI when generating

[jira] [Updated] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs

2017-01-11 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19182: - Description: When DStreamGraph is generating a job, it will hold a lock and block other APIs.

[jira] [Created] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs

2017-01-11 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-19182: Summary: Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs Key: SPARK-19182 URL: https://issues.apache.org/jira/browse/SPARK-19182

[jira] [Resolved] (SPARK-19132) Add test cases for row size estimation

2017-01-11 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-19132. - Resolution: Fixed Fix Version/s: 2.2.0 > Add test cases for row size estimation >

[jira] [Commented] (SPARK-18823) Assignation by column name variable not available or bug?

2017-01-11 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819446#comment-15819446 ] Shivaram Venkataraman commented on SPARK-18823: --- Yeah I think it makes sense to not handle

[jira] [Commented] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-11 Thread yucai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819400#comment-15819400 ] yucai commented on SPARK-19180: --- Hi Owen, Thanks a lot for comments, it is using unsafe API for

[jira] [Resolved] (SPARK-18801) Support resolve a nested view

2017-01-11 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18801. --- Resolution: Fixed Assignee: Jiang Xingbo > Support resolve a nested view >

[jira] [Resolved] (SPARK-19130) SparkR should support setting and adding new column with singular value implicitly

2017-01-11 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-19130. --- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-19177) SparkR Data Frame operation between columns elements

2017-01-11 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819260#comment-15819260 ] Shivaram Venkataraman commented on SPARK-19177: --- Thanks [~masip85] - Can you include a

[jira] [Commented] (SPARK-19181) SparkListenerSuite.local metrics fails when average executorDeserializeTime is too short.

2017-01-11 Thread Jose Soltren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819186#comment-15819186 ] Jose Soltren commented on SPARK-19181: -- SPARK-2208 disabled a similar metric previously. >

[jira] [Created] (SPARK-19181) SparkListenerSuite.local metrics fails when average executorDeserializeTime is too short.

2017-01-11 Thread Jose Soltren (JIRA)
Jose Soltren created SPARK-19181: Summary: SparkListenerSuite.local metrics fails when average executorDeserializeTime is too short. Key: SPARK-19181 URL: https://issues.apache.org/jira/browse/SPARK-19181

[jira] [Assigned] (SPARK-9435) Java UDFs don't work with GROUP BY expressions

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9435: --- Assignee: Apache Spark > Java UDFs don't work with GROUP BY expressions >

[jira] [Assigned] (SPARK-9435) Java UDFs don't work with GROUP BY expressions

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-9435: --- Assignee: (was: Apache Spark) > Java UDFs don't work with GROUP BY expressions >

[jira] [Commented] (SPARK-9435) Java UDFs don't work with GROUP BY expressions

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819117#comment-15819117 ] Apache Spark commented on SPARK-9435: - User 'HyukjinKwon' has created a pull request for this issue:

[jira] [Commented] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819073#comment-15819073 ] Sean Owen commented on SPARK-19180: --- Are you sure? most stuff is int aligned in the JVM. You might be

[jira] [Commented] (SPARK-17136) Design optimizer interface for ML algorithms

2017-01-11 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819062#comment-15819062 ] Seth Hendrickson commented on SPARK-17136: -- I'm interested in working on this task including

[jira] [Created] (SPARK-19180) the offset of short is 4 in OffHeapColumnVector's putShorts

2017-01-11 Thread yucai (JIRA)
yucai created SPARK-19180: - Summary: the offset of short is 4 in OffHeapColumnVector's putShorts Key: SPARK-19180 URL: https://issues.apache.org/jira/browse/SPARK-19180 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-17568) Add spark-submit option for user to override ivy settings used to resolve packages/artifacts

2017-01-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17568. Resolution: Fixed Assignee: Bryan Cutler Fix Version/s: 2.2.0 > Add

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-01-11 Thread Dan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818849#comment-15818849 ] Dan commented on SPARK-18075: - If he is running into the same issue with spark-shell, which is one of the

[jira] [Assigned] (SPARK-19152) DataFrameWriter.saveAsTable should work with hive format with append mode

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19152: Assignee: Apache Spark > DataFrameWriter.saveAsTable should work with hive format with

[jira] [Commented] (SPARK-19152) DataFrameWriter.saveAsTable should work with hive format with append mode

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818731#comment-15818731 ] Apache Spark commented on SPARK-19152: -- User 'windpiger' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19152) DataFrameWriter.saveAsTable should work with hive format with append mode

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19152: Assignee: (was: Apache Spark) > DataFrameWriter.saveAsTable should work with hive

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-01-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818720#comment-15818720 ] Sean Owen commented on SPARK-18075: --- Yes, spark-shell is submitted the same way. If you wrote some code

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-01-11 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818716#comment-15818716 ] nirav patel commented on SPARK-19090: - [~q79969786] As I mentioned in previous comment it does work

[jira] [Commented] (SPARK-17101) Provide consistent format identifiers for TextFileFormat and ParquetFileFormat

2017-01-11 Thread Shuai Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818648#comment-15818648 ] Shuai Lin commented on SPARK-17101: --- Seems this issue has already been resolved by

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-01-11 Thread Nick Orka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818645#comment-15818645 ] Nick Orka commented on SPARK-18075: --- This is really cool conversation, but how about if I run it in

[jira] [Created] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19179: - Summary: spark.yarn.access.namenodes description is wrong Key: SPARK-19179 URL: https://issues.apache.org/jira/browse/SPARK-19179 Project: Spark Issue

[jira] [Commented] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files

2017-01-11 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818635#comment-15818635 ] roncenzhao commented on SPARK-19169: I have the two doubts: 1. In the method

[jira] [Resolved] (SPARK-19021) Generailize HDFSCredentialProvider to support non HDFS security FS

2017-01-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19021. --- Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.2.0 > Generailize

[jira] [Commented] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files

2017-01-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818599#comment-15818599 ] Sean Owen commented on SPARK-19169: --- It sounds like you're saying you read the data with the wrong

[jira] [Commented] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.

2017-01-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818594#comment-15818594 ] Sean Owen commented on SPARK-13198: --- I think that's up to you if you're interested in this? it's not

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-01-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818591#comment-15818591 ] Sean Owen commented on SPARK-18075: --- It's possible in many cases already and always has been.

[jira] [Commented] (SPARK-19169) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files

2017-01-11 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818552#comment-15818552 ] roncenzhao commented on SPARK-19169: I do not think this is a misusage of ORC. If we do not set the

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-01-11 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818540#comment-15818540 ] Wenchen Fan commented on SPARK-18075: - Although it's not a bug, I think this could be a very cool

[jira] [Commented] (SPARK-18075) UDF doesn't work on non-local spark

2017-01-11 Thread Michael David Pedersen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818526#comment-15818526 ] Michael David Pedersen commented on SPARK-18075: I'm encountering this problem too, in

[jira] [Comment Edited] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.

2017-01-11 Thread Dmytro Bielievtsov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818498#comment-15818498 ] Dmytro Bielievtsov edited comment on SPARK-13198 at 1/11/17 2:38 PM: -

[jira] [Commented] (SPARK-13198) sc.stop() does not clean up on driver, causes Java heap OOM.

2017-01-11 Thread Dmytro Bielievtsov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818498#comment-15818498 ] Dmytro Bielievtsov commented on SPARK-13198: [~srowen] Looks like a growing number of people

[jira] [Commented] (SPARK-19132) Add test cases for row size estimation

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818492#comment-15818492 ] Apache Spark commented on SPARK-19132: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19132) Add test cases for row size estimation

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19132: Assignee: Zhenhua Wang (was: Apache Spark) > Add test cases for row size estimation >

[jira] [Assigned] (SPARK-19178) convert string of large numbers to int should return null

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19178: Assignee: Wenchen Fan (was: Apache Spark) > convert string of large numbers to int

[jira] [Assigned] (SPARK-19132) Add test cases for row size estimation

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19132: Assignee: Apache Spark (was: Zhenhua Wang) > Add test cases for row size estimation >

[jira] [Commented] (SPARK-19178) convert string of large numbers to int should return null

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818477#comment-15818477 ] Apache Spark commented on SPARK-19178: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19178) convert string of large numbers to int should return null

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19178: Assignee: Apache Spark (was: Wenchen Fan) > convert string of large numbers to int

[jira] [Created] (SPARK-19178) convert string of large numbers to int should return null

2017-01-11 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-19178: --- Summary: convert string of large numbers to int should return null Key: SPARK-19178 URL: https://issues.apache.org/jira/browse/SPARK-19178 Project: Spark

[jira] [Commented] (SPARK-19151) DataFrameWriter.saveAsTable should work with hive format with overwrite mode

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818366#comment-15818366 ] Apache Spark commented on SPARK-19151: -- User 'windpiger' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19151) DataFrameWriter.saveAsTable should work with hive format with overwrite mode

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19151: Assignee: Apache Spark > DataFrameWriter.saveAsTable should work with hive format with

[jira] [Assigned] (SPARK-19151) DataFrameWriter.saveAsTable should work with hive format with overwrite mode

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19151: Assignee: (was: Apache Spark) > DataFrameWriter.saveAsTable should work with hive

[jira] [Commented] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files

2017-01-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818344#comment-15818344 ] Sean Owen commented on SPARK-19175: --- No, continue on the JIRA I left open, SPARK-19169. This sounds

[jira] [Commented] (SPARK-19158) ml.R example fails in yarn-cluster mode due to lacks of e1071 package

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818300#comment-15818300 ] Apache Spark commented on SPARK-19158: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19158) ml.R example fails in yarn-cluster mode due to lacks of e1071 package

2017-01-11 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19158: Assignee: (was: Apache Spark) > ml.R example fails in yarn-cluster mode due to lacks

  1   2   >