[jira] [Comment Edited] (SPARK-11319) PySpark silently Accepts null values in non-nullable DataFrame fields.

[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014098#comment-15014098 ] Harry Brundage edited comment on SPARK-11319 at 11/19/15 6:30 PM: --

[jira] [Commented] (SPARK-11319) PySpark silently Accepts null values in non-nullable DataFrame fields.

[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014098#comment-15014098 ] Harry Brundage commented on SPARK-11319: Forgive my frankness but that is ridiculous. This means

[jira] [Commented] (SPARK-11319) PySpark silently Accepts null values in non-nullable DataFrame fields.

[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014405#comment-15014405 ] Harry Brundage commented on SPARK-11319: Why force all users everywhere to do a pass over the

[jira] [Commented] (SPARK-11319) PySpark silently Accepts null values in non-nullable DataFrame fields.

[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014414#comment-15014414 ] Harry Brundage commented on SPARK-11319: I also don't think that "JSON data is likely to have

[jira] [Commented] (SPARK-11319) PySpark silently Accepts null values in non-nullable DataFrame fields.

[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014711#comment-15014711 ] Harry Brundage commented on SPARK-11319: Oh I forgot to mention I could imagine us settling on a

[jira] [Commented] (SPARK-11319) PySpark silently Accepts null values in non-nullable DataFrame fields.

[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014705#comment-15014705 ] Harry Brundage commented on SPARK-11319: bq. First of all, lets talk about nullability in the

[jira] [Commented] (SPARK-4389) Set akka.remote.netty.tcp.bind-hostname=0.0.0.0 so driver can be located behind NAT

[ https://issues.apache.org/jira/browse/SPARK-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600206#comment-14600206 ] Harry Brundage commented on SPARK-4389: --- Any movement on this? Is akka 2.4 happening

[jira] [Comment Edited] (SPARK-7009) Build assembly JAR via ant to avoid zip64 problems

[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589913#comment-14589913 ] Harry Brundage edited comment on SPARK-7009 at 6/17/15 3:22 PM:

[jira] [Updated] (SPARK-7009) Build assembly JAR via ant to avoid zip64 problems

[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-7009: -- Attachment: check_spark_python.sh Script for checking to see if a spark release artifact jar

[jira] [Commented] (SPARK-7009) Build assembly JAR via ant to avoid zip64 problems

[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589913#comment-14589913 ] Harry Brundage commented on SPARK-7009: --- [~sowen] I wrote a script to test that we

[jira] [Commented] (SPARK-7009) Build assembly JAR via ant to avoid zip64 problems

[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589004#comment-14589004 ] Harry Brundage commented on SPARK-7009: --- We're still experiencing this issue with

[jira] [Closed] (SPARK-4732) All application progress on the standalone scheduler can be halted by one systematically faulty node

[ https://issues.apache.org/jira/browse/SPARK-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage closed SPARK-4732. - Resolution: Duplicate All application progress on the standalone scheduler can be halted by one

[jira] [Updated] (SPARK-6917) Broken data returned to PySpark dataframe if any large numbers used in Scala land

[ https://issues.apache.org/jira/browse/SPARK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-6917: -- Description: When trying to access data stored in a Parquet file with an INT96 column (read:

[jira] [Commented] (SPARK-6917) Broken data returned to PySpark dataframe if any large numbers used in Scala land

[ https://issues.apache.org/jira/browse/SPARK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503915#comment-14503915 ] Harry Brundage commented on SPARK-6917: --- [~davies] or [~joshrosen] any idea why this

[jira] [Created] (SPARK-6917) Broken data returned to PySpark dataframe if any large numbers used in Scala land

Harry Brundage created SPARK-6917: - Summary: Broken data returned to PySpark dataframe if any large numbers used in Scala land Key: SPARK-6917 URL: https://issues.apache.org/jira/browse/SPARK-6917

[jira] [Updated] (SPARK-6917) Broken data returned to PySpark dataframe if any large numbers used in Scala land

[ https://issues.apache.org/jira/browse/SPARK-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-6917: -- Attachment: part-r-1.parquet Attached is the parquet file I used. Broken data returned

[jira] [Commented] (SPARK-6411) PySpark DataFrames can't be created if any datetimes have timezones

[ https://issues.apache.org/jira/browse/SPARK-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14379947#comment-14379947 ] Harry Brundage commented on SPARK-6411: --- I've opened and issue on the upstream

[jira] [Created] (SPARK-6411) PySpark DataFrames can't be created if any datetimes have timezones

Harry Brundage created SPARK-6411: - Summary: PySpark DataFrames can't be created if any datetimes have timezones Key: SPARK-6411 URL: https://issues.apache.org/jira/browse/SPARK-6411 Project: Spark

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370598#comment-14370598 ] Harry Brundage commented on SPARK-4105: --- Would just like to add that we are seeing

[jira] [Comment Edited] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370598#comment-14370598 ] Harry Brundage edited comment on SPARK-4105 at 3/20/15 2:46 AM:

[jira] [Commented] (SPARK-2688) Need a way to run multiple data pipeline concurrently

[ https://issues.apache.org/jira/browse/SPARK-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289771#comment-14289771 ] Harry Brundage commented on SPARK-2688: --- I respectfully disagree :) Persist is one

[jira] [Commented] (SPARK-5162) Python yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270543#comment-14270543 ] Harry Brundage commented on SPARK-5162: --- [~sandyr] are you familiar with why the

[jira] [Commented] (SPARK-4732) All application progress on the standalone scheduler can be halted by one systematically faulty node

[ https://issues.apache.org/jira/browse/SPARK-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248367#comment-14248367 ] Harry Brundage commented on SPARK-4732: --- Seems like it would, feel free to mark as

[jira] [Created] (SPARK-4732) All application progress on the standalone scheduler can be halted by one systematically faulty node

Harry Brundage created SPARK-4732: - Summary: All application progress on the standalone scheduler can be halted by one systematically faulty node Key: SPARK-4732 URL:

[jira] [Updated] (SPARK-4732) All application progress on the standalone scheduler can be halted by one systematically faulty node

[ https://issues.apache.org/jira/browse/SPARK-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-4732: -- Description: We've experienced several cluster wide outages caused by unexpected system wide

[jira] [Updated] (SPARK-4732) All application progress on the standalone scheduler can be halted by one systematically faulty node

[ https://issues.apache.org/jira/browse/SPARK-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-4732: -- Description: We've experienced several cluster wide outages caused by unexpected system wide

[jira] [Updated] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

[ https://issues.apache.org/jira/browse/SPARK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-4498: -- Attachment: one-applications-master-logs.txt These are the logs from the standalone master for

[jira] [Updated] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

[ https://issues.apache.org/jira/browse/SPARK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-4498: -- Description: We observe the spark standalone master not detecting that a driver application

[jira] [Created] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

Harry Brundage created SPARK-4498: - Summary: Standalone Master can fail to recognize completed/failed applications Key: SPARK-4498 URL: https://issues.apache.org/jira/browse/SPARK-4498 Project: Spark

[jira] [Updated] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

[ https://issues.apache.org/jira/browse/SPARK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-4498: -- Attachment: all-master-logs-around-blip.txt These are all the master logs around a recent blip

[jira] [Commented] (SPARK-4498) Standalone Master can fail to recognize completed/failed applications

[ https://issues.apache.org/jira/browse/SPARK-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218491#comment-14218491 ] Harry Brundage commented on SPARK-4498: --- For the simple canary spark application

[jira] [Updated] (SPARK-1849) Broken UTF-8 encoded data gets character replacements and thus can't be fixed

[ https://issues.apache.org/jira/browse/SPARK-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harry Brundage updated SPARK-1849: -- Attachment: encoding_test Here's the windows encoded file I was using to test with if you'd

[jira] [Created] (SPARK-1849) Broken UTF-8 encoded data gets character replacements and thus can't be fixed

Harry Brundage created SPARK-1849: - Summary: Broken UTF-8 encoded data gets character replacements and thus can't be fixed Key: SPARK-1849 URL: https://issues.apache.org/jira/browse/SPARK-1849

[jira] [Comment Edited] (SPARK-1849) Broken UTF-8 encoded data gets character replacements and thus can't be fixed

[ https://issues.apache.org/jira/browse/SPARK-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14000460#comment-14000460 ] Harry Brundage edited comment on SPARK-1849 at 5/16/14 11:02 PM:

[jira] [Commented] (SPARK-1849) Broken UTF-8 encoded data gets character replacements and thus can't be fixed

[ https://issues.apache.org/jira/browse/SPARK-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14000460#comment-14000460 ] Harry Brundage commented on SPARK-1849: --- I disagree - the data isn't badly encoded,

[jira] [Commented] (SPARK-791) [pyspark] operator.getattr not serialized

[ https://issues.apache.org/jira/browse/SPARK-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988698#comment-13988698 ] Harry Brundage commented on SPARK-791: -- Hey Josh, looks like that issue mentioned in