[jira] [Commented] (SPARK-26439) Introduce WorkerOffer reservation mechanism for Barrier TaskSet

2019-01-07 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736687#comment-16736687 ] wuyi commented on SPARK-26439: -- [~srowen] ok. > Introduce WorkerOffer reservation mechanis

[jira] [Commented] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread eaton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736685#comment-16736685 ] eaton commented on SPARK-26567: --- Ok. [~hyukjin.kwon] > Should we align CSV query results

[jira] [Updated] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread eaton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eaton updated SPARK-26567: -- Description: If we want to be consistent, we can modify the makeConverter function in UnivocityParser, but th

[jira] [Commented] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736683#comment-16736683 ] Hyukjin Kwon commented on SPARK-26567: -- I don't think we should necessarily follow

[jira] [Resolved] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26567. -- Resolution: Won't Fix > Should we align CSV query results with hive text query results: an int

[jira] [Updated] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread eaton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eaton updated SPARK-26567: -- Description: If we want to be consistent, we can modify the makeConverter function in UnivocityParser, but th

[jira] [Updated] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread eaton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eaton updated SPARK-26567: -- Description: If we want to be consistent, we can modify the makeConverter function in UnivocityParser, but th

[jira] [Updated] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread eaton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eaton updated SPARK-26567: -- Description: If we want to be consistent, we can modify the makeConverter function in UnivocityParser, but th

[jira] [Created] (SPARK-26567) Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null

2019-01-07 Thread eaton (JIRA)
eaton created SPARK-26567: - Summary: Should we align CSV query results with hive text query results: an int field, if the input value is 1.0, hive text query results is 1, CSV query results is null Key: SPARK-26567 URL:

[jira] [Resolved] (SPARK-26562) countDistinct and user-defined function cannot be used in SELECT

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26562. -- Resolution: Cannot Reproduce I can't reproduce this: {code} >>> df.select(F.sum(df['a']), F.c

[jira] [Commented] (SPARK-26558) java.util.NoSuchElementException while saving data into HDFS using Spark

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736662#comment-16736662 ] Hyukjin Kwon commented on SPARK-26558: -- The exception itself is simply from {{green

[jira] [Resolved] (SPARK-24196) Spark Thrift Server - SQL Client connections does't show db artefacts

2019-01-07 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24196. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 3.0.0 > Spark Thrift Server - SQL

[jira] [Resolved] (SPARK-26558) java.util.NoSuchElementException while saving data into HDFS using Spark

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26558. -- Resolution: Invalid > java.util.NoSuchElementException while saving data into HDFS using Spark

[jira] [Updated] (SPARK-26431) Update availableSlots by availableCpus for barrier taskset

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26431: -- Target Version/s: (was: 2.4.0) > Update availableSlots by availableCpus for barrier taskset > --

[jira] [Updated] (SPARK-24464) Unit tests for MLlib's Instrumentation

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24464: -- Target Version/s: (was: 2.4.0) > Unit tests for MLlib's Instrumentation > --

[jira] [Resolved] (SPARK-23693) SQL function uuid()

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23693. --- Resolution: Won't Fix > SQL function uuid() > --- > > Key: SPARK-236

[jira] [Commented] (SPARK-26543) Support the coordinator to determine post-shuffle partitions more reasonably

2019-01-07 Thread chenliang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736653#comment-16736653 ] chenliang commented on SPARK-26543: --- [~yhuai] [~aaronmarkham]  Could you please have a

[jira] [Updated] (SPARK-26439) Introduce WorkerOffer reservation mechanism for Barrier TaskSet

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26439: -- Flags: (was: Important) Target Version/s: (was: 2.4.0) Fix Version/s: (wa

[jira] [Updated] (SPARK-25346) Document Spark builtin data sources

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25346: -- Target Version/s: (was: 2.4.0) > Document Spark builtin data sources > -

[jira] [Updated] (SPARK-25584) Document libsvm data source in doc site

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25584: -- Target Version/s: (was: 2.4.0) > Document libsvm data source in doc site > -

[jira] [Resolved] (SPARK-26334) NullPointerException in CallMethodViaReflection when we apply reflect function for empty field

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26334. --- Resolution: Not A Problem > NullPointerException in CallMethodViaReflection when we apply reflect

[jira] [Commented] (SPARK-25823) map_filter can generate incorrect data

2019-01-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736645#comment-16736645 ] Dongjoon Hyun commented on SPARK-25823: --- +1. Thank you, [~hyukjin.kwon]. > map_fi

[jira] [Comment Edited] (SPARK-25823) map_filter can generate incorrect data

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736582#comment-16736582 ] Hyukjin Kwon edited comment on SPARK-25823 at 1/8/19 2:09 AM:

[jira] [Commented] (SPARK-25823) map_filter can generate incorrect data

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736582#comment-16736582 ] Hyukjin Kwon commented on SPARK-25823: -- Looks [~Thincrs] bot is still active. I'm g

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736580#comment-16736580 ] Wenchen Fan commented on SPARK-26565: - The release tools have been evolved. In the b

[jira] [Commented] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736540#comment-16736540 ] Bryan Cutler commented on SPARK-26566: -- Version 0.12.0 is slated to be released in

[jira] [Resolved] (SPARK-23386) Enable direct application links before replay

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23386. Resolution: Won't Fix I'm closing this for now as I believe SPARK-6951 handles a lot of th

[jira] [Updated] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-26566: - Description: _This is just a placeholder for now to collect what needs to be fixed when we upgr

[jira] [Resolved] (SPARK-26267) Kafka source may reprocess data

2019-01-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-26267. -- Resolution: Fixed Fix Version/s: 2.4.1 > Kafka source may reprocess data >

[jira] [Updated] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-26566: - Affects Version/s: (was: 2.3.0) 2.4.0 > Upgrade apache/arrow to 0.12.

[jira] [Updated] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-26566: - Fix Version/s: (was: 2.4.0) > Upgrade apache/arrow to 0.12.0 > -

[jira] [Assigned] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-26566: Assignee: (was: Bryan Cutler) > Upgrade apache/arrow to 0.12.0 >

[jira] [Updated] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-26566: - Target Version/s: (was: 2.4.0) > Upgrade apache/arrow to 0.12.0 >

[jira] [Created] (SPARK-26566) Upgrade apache/arrow to 0.12.0

2019-01-07 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-26566: Summary: Upgrade apache/arrow to 0.12.0 Key: SPARK-26566 URL: https://issues.apache.org/jira/browse/SPARK-26566 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-25272) Show some kind of test output to indicate pyarrow tests were run

2019-01-07 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-25272. -- Resolution: Won't Fix > Show some kind of test output to indicate pyarrow tests were run > ---

[jira] [Issue Comment Deleted] (SPARK-25823) map_filter can generate incorrect data

2019-01-07 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25823: -- Comment: was deleted (was: A user of thincrs has selected this issue. Deadline: Mon, Jan 14, 2

[jira] [Comment Edited] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736500#comment-16736500 ] shane knapp edited comment on SPARK-26565 at 1/7/19 11:57 PM:

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736500#comment-16736500 ] shane knapp commented on SPARK-26565: - [~vanzin] – good to have confirmed that `do-r

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736493#comment-16736493 ] Marcelo Vanzin commented on SPARK-26565: Stepping back a little, what is the pur

[jira] [Assigned] (SPARK-26491) Use ConfigEntry for hardcoded configs for test categories.

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26491: -- Assignee: Marco Gaido > Use ConfigEntry for hardcoded configs for test categories. >

[jira] [Resolved] (SPARK-26491) Use ConfigEntry for hardcoded configs for test categories.

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26491. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23413 [https:

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736485#comment-16736485 ] Sean Owen commented on SPARK-26565: --- I've never done a release myself ... [~cloud_fan]

[jira] [Comment Edited] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736472#comment-16736472 ] shane knapp edited comment on SPARK-26565 at 1/7/19 11:26 PM:

[jira] [Updated] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26565: Attachment: no-idea.jpg > modify dev/create-release/release-build.sh to let jenkins build packages

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736472#comment-16736472 ] shane knapp commented on SPARK-26565: - [~srowen] – herein lies my confusion...  afte

[jira] [Updated] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26565: Description: about a year+ ago, we stopped publishing releases directly from jenkins... this mean

[jira] [Commented] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736465#comment-16736465 ] Sean Owen commented on SPARK-26565: --- Sure, if we're not making releases with it, no ne

[jira] [Assigned] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26565: Assignee: shane knapp (was: Apache Spark) > modify dev/create-release/release-build.sh t

[jira] [Assigned] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26565: Assignee: Apache Spark (was: shane knapp) > modify dev/create-release/release-build.sh t

[jira] [Updated] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26565: Description: about a year+ ago, we stopped publishing releases directly from jenkins... this mean

[jira] [Created] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
shane knapp created SPARK-26565: --- Summary: modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing Key: SPARK-26565 URL: https://issues.apache.org/jira/browse/SPARK-26565

[jira] [Updated] (SPARK-26565) modify dev/create-release/release-build.sh to let jenkins build packages w/o publishing

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-26565: Attachment: fine.png > modify dev/create-release/release-build.sh to let jenkins build packages w/

[jira] [Updated] (SPARK-26269) YarnAllocator should have same blacklist behaviour with YARN to maxmize use of cluster resource

2019-01-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-26269: -- Fix Version/s: 2.4.1 > YarnAllocator should have same blacklist behaviour with YARN to maxmize

[jira] [Assigned] (SPARK-26564) Fix misleading error message about spark.network.timeout and spark.executor.heartbeatInterval

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26564: Assignee: (was: Apache Spark) > Fix misleading error message about spark.network.time

[jira] [Assigned] (SPARK-26564) Fix misleading error message about spark.network.timeout and spark.executor.heartbeatInterval

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26564: Assignee: Apache Spark > Fix misleading error message about spark.network.timeout and >

[jira] [Commented] (SPARK-25823) map_filter can generate incorrect data

2019-01-07 Thread Thincrs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736446#comment-16736446 ] Thincrs commented on SPARK-25823: - A user of thincrs has selected this issue. Deadline:

[jira] [Created] (SPARK-26564) Fix misleading error message about spark.network.timeout and spark.executor.heartbeatInterval

2019-01-07 Thread Kengo Seki (JIRA)
Kengo Seki created SPARK-26564: -- Summary: Fix misleading error message about spark.network.timeout and spark.executor.heartbeatInterval Key: SPARK-26564 URL: https://issues.apache.org/jira/browse/SPARK-26564

[jira] [Closed] (SPARK-26537) update the release scripts to point to gitbox

2019-01-07 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp closed SPARK-26537. --- > update the release scripts to point to gitbox > - > >

[jira] [Resolved] (SPARK-26065) Change query hint from a `LogicalPlan` to a field

2019-01-07 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-26065. - Resolution: Fixed Assignee: Maryann Xue Fix Version/s: 3.0.0 > Change query hint from a

[jira] [Commented] (SPARK-24701) SparkMaster WebUI allow all appids to be shown in detail on port 4040 rather than different ports per app

2019-01-07 Thread Pablo Langa Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736366#comment-16736366 ] Pablo Langa Blanco commented on SPARK-24701: Hi [~toopt4] i think you have

[jira] [Created] (SPARK-26563) Quick Start documentation provides example that doesn't work (Java)

2019-01-07 Thread Leonardo Colman Lopes (JIRA)
Leonardo Colman Lopes created SPARK-26563: - Summary: Quick Start documentation provides example that doesn't work (Java) Key: SPARK-26563 URL: https://issues.apache.org/jira/browse/SPARK-26563

[jira] [Assigned] (SPARK-26563) Quick Start documentation provides example that doesn't work (Java)

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26563: Assignee: (was: Apache Spark) > Quick Start documentation provides example that doesn

[jira] [Commented] (SPARK-26563) Quick Start documentation provides example that doesn't work (Java)

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736363#comment-16736363 ] Apache Spark commented on SPARK-26563: -- User 'Kerooker' has created a pull request

[jira] [Assigned] (SPARK-26563) Quick Start documentation provides example that doesn't work (Java)

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26563: Assignee: Apache Spark > Quick Start documentation provides example that doesn't work (Ja

[jira] [Commented] (SPARK-26563) Quick Start documentation provides example that doesn't work (Java)

2019-01-07 Thread Leonardo Colman Lopes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736358#comment-16736358 ] Leonardo Colman Lopes commented on SPARK-26563: --- Pull request ready https

[jira] [Resolved] (SPARK-25689) Move token renewal logic to driver in yarn-client mode

2019-01-07 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-25689. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23338 [https://gi

[jira] [Assigned] (SPARK-25689) Move token renewal logic to driver in yarn-client mode

2019-01-07 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-25689: Assignee: Marcelo Vanzin > Move token renewal logic to driver in yarn-client mode > -

[jira] [Updated] (SPARK-26551) Selecting one complex field and having is null predicate on another complex field can cause error

2019-01-07 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-26551: Affects Version/s: 2.4.0 > Selecting one complex field and having is null predicate on another complex >

[jira] [Assigned] (SPARK-26551) Selecting one complex field and having is null predicate on another complex field can cause error

2019-01-07 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-26551: --- Assignee: Liang-Chi Hsieh > Selecting one complex field and having is null predicate on another com

[jira] [Commented] (SPARK-26518) UI Application Info Race Condition Can Throw NoSuchElement

2019-01-07 Thread Russell Spitzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736091#comment-16736091 ] Russell Spitzer commented on SPARK-26518: - Yeah I basically came to the same con

[jira] [Resolved] (SPARK-26561) Symbol 'type org.apache.spark.Logging' is missing from the classpath. This symbol is required by 'class org.apache.hadoop.hbase.spark.HBaseContext'

2019-01-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26561. Resolution: Not A Bug External libraries should not use internal Spark classes like {{Logg

[jira] [Commented] (SPARK-26518) UI Application Info Race Condition Can Throw NoSuchElement

2019-01-07 Thread Pablo Langa Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736087#comment-16736087 ] Pablo Langa Blanco commented on SPARK-26518: i was trying to find and easy s

[jira] [Resolved] (SPARK-24489) No check for invalid input type of weight data in ml.PowerIterationClustering

2019-01-07 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-24489. - Resolution: Fixed Fix Version/s: 3.0.0 Thank's for working on this, I've merged the fix into mast

[jira] [Assigned] (SPARK-24489) No check for invalid input type of weight data in ml.PowerIterationClustering

2019-01-07 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-24489: --- Assignee: shahid > No check for invalid input type of weight data in ml.PowerIterationClustering >

[jira] [Commented] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-07 Thread Pablo Langa Blanco (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736024#comment-16736024 ] Pablo Langa Blanco commented on SPARK-26457: Ok, I don't know that you are w

[jira] [Commented] (SPARK-26225) Scan: track decoding time for row-based data sources

2019-01-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735954#comment-16735954 ] Wenchen Fan commented on SPARK-26225: - I think it's hard to define the decoding time

[jira] [Assigned] (SPARK-26383) NPE when use DataFrameReader.jdbc with wrong URL

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26383: - Assignee: clouds > NPE when use DataFrameReader.jdbc with wrong URL > -

[jira] [Commented] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2019-01-07 Thread abhinav (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735942#comment-16735942 ] abhinav commented on SPARK-13928: - Thanks a lot Wen > Move org.apache.spark.Logging

[jira] [Resolved] (SPARK-26513) Trigger GC on executor node idle

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26513. --- Resolution: Won't Fix > Trigger GC on executor node idle > > >

[jira] [Resolved] (SPARK-26383) NPE when use DataFrameReader.jdbc with wrong URL

2019-01-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26383. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23464 [https://github.c

[jira] [Updated] (SPARK-26549) PySpark worker reuse take no effect for parallelize xrange

2019-01-07 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-26549: Description: During [the follow-up work|https://github.com/apache/spark/pull/23435#issuecomment-4

[jira] [Commented] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2019-01-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735862#comment-16735862 ] Wenchen Fan commented on SPARK-13928: - You need to check the HBase document and see

[jira] [Commented] (SPARK-24725) Discuss necessary info and access in barrier mode + Mesos

2019-01-07 Thread Angel Conde (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735836#comment-16735836 ] Angel Conde commented on SPARK-24725: -- Mesos has already the possibility via MPI f

[jira] [Updated] (SPARK-26549) PySpark worker reuse take no effect for parallelize xrange

2019-01-07 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-26549: Description: During [the follow-up work|https://github.com/apache/spark/pull/23435#issuecomment-4

[jira] [Updated] (SPARK-26549) PySpark worker reuse take no effect for parallelize xrange

2019-01-07 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-26549: Description: During [the follow-up work|https://github.com/apache/spark/pull/23435#issuecomment-4

[jira] [Updated] (SPARK-26549) PySpark worker reuse take no effect for parallelize xrange

2019-01-07 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-26549: Description: During [the follow-up work|https://github.com/apache/spark/pull/23435#issuecomment-4

[jira] [Created] (SPARK-26562) countDistinct and user-defined function cannot be used in SELECT

2019-01-07 Thread Ravi Kaushik (JIRA)
Ravi Kaushik created SPARK-26562: Summary: countDistinct and user-defined function cannot be used in SELECT Key: SPARK-26562 URL: https://issues.apache.org/jira/browse/SPARK-26562 Project: Spark

[jira] [Updated] (SPARK-26549) PySpark worker reuse take no effect for parallelize xrange

2019-01-07 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-26549: Description: During [the follow-up work|https://github.com/apache/spark/pull/23435#issuecomment-4

[jira] [Updated] (SPARK-26549) PySpark worker reuse take no effect for parallelize xrange

2019-01-07 Thread Yuanjian Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-26549: Summary: PySpark worker reuse take no effect for parallelize xrange (was: PySpark worker reuse ta

[jira] [Assigned] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26457: Assignee: Apache Spark > Show hadoop configurations in HistoryServer environment tab > --

[jira] [Resolved] (SPARK-23636) [SPARK 2.2] | Kafka Consumer | KafkaUtils.createRDD throws Exception - java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

2019-01-07 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-23636. --- Resolution: Fixed Fix Version/s: 2.4.0 Please reopen if problem re-appears. > [SPARK

[jira] [Assigned] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-07 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26457: Assignee: (was: Apache Spark) > Show hadoop configurations in HistoryServer environme

[jira] [Commented] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2019-01-07 Thread abhinav (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735744#comment-16735744 ] abhinav commented on SPARK-13928: - Wen, I have already included the "org.apache.spark.Lo

[jira] [Commented] (SPARK-26457) Show hadoop configurations in HistoryServer environment tab

2019-01-07 Thread deshanxiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735707#comment-16735707 ] deshanxiao commented on SPARK-26457: [~planga82] Hi, thanks for your reply! I know t

[jira] [Updated] (SPARK-26534) Closure Cleaner Bug

2019-01-07 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam updated SPARK-26534: Description: I've found a strange combination of closures where the closure cleaner doesn't seem to be smart enou

[jira] [Commented] (SPARK-26534) Closure Cleaner Bug

2019-01-07 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735699#comment-16735699 ] sam commented on SPARK-26534: - [~viirya] If I change to RDD I cannot reproduce either.  Thi

[jira] [Commented] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2019-01-07 Thread abhinav (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735691#comment-16735691 ] abhinav commented on SPARK-13928: - Hi, I am using HBase 1.2.0. is that fine? Can you te

[jira] [Commented] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2019-01-07 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735665#comment-16735665 ] Wenchen Fan commented on SPARK-13928: - what version of HBase connector you are using

[jira] [Created] (SPARK-26561) Symbol 'type org.apache.spark.Logging' is missing from the classpath. This symbol is required by 'class org.apache.hadoop.hbase.spark.HBaseContext'

2019-01-07 Thread abhinav (JIRA)
abhinav created SPARK-26561: --- Summary: Symbol 'type org.apache.spark.Logging' is missing from the classpath. This symbol is required by 'class org.apache.hadoop.hbase.spark.HBaseContext' Key: SPARK-26561 URL: https://i

[jira] [Updated] (SPARK-26559) ML image can't work with numpy versions prior to 1.9

2019-01-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26559: - Fix Version/s: 3.0.0 2.4.1 > ML image can't work with numpy versions prior to

  1   2   >