[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-11-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263690#comment-16263690 ] Hyukjin Kwon commented on SPARK-22240: -- Sure, sounds good ! > S3 CSV number of part

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-11-16 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255861#comment-16255861 ] Steve Loughran commented on SPARK-22240: I think there are two separate issues an

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223564#comment-16223564 ] Hyukjin Kwon commented on SPARK-22240: -- I am sorry, I have been super busy these for

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-28 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223400#comment-16223400 ] Steve Loughran commented on SPARK-22240: so this partition calculation problem is

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217398#comment-16217398 ] Steve Loughran commented on SPARK-22240: no, spark 2.2 doesn't fix this. I have

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-24 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16217325#comment-16217325 ] Steve Loughran commented on SPARK-22240: I'm doing some testing with master & rea

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204019#comment-16204019 ] Hyukjin Kwon commented on SPARK-22240: -- Here, I could reproduce the issue in the JIR

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203966#comment-16203966 ] Steve Loughran commented on SPARK-22240: Point me at a simple test suite for the

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203771#comment-16203771 ] Hyukjin Kwon commented on SPARK-22240: -- Yea, this problem exists when {{multiLine}}.

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203709#comment-16203709 ] Steve Loughran commented on SPARK-22240: [~hyukjin.kwon]: we now see that on s3a,

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203600#comment-16203600 ] Hyukjin Kwon commented on SPARK-22240: -- I am sorry for late response. {quote} If i

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-13 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203495#comment-16203495 ] Steve Loughran commented on SPARK-22240: We've got a test in HADOOP-14943 which l

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-12 Thread Arthur Baudry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202691#comment-16202691 ] Arthur Baudry commented on SPARK-22240: --- [~hyukjin.kwon] Yes it is a single file so

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201863#comment-16201863 ] Steve Loughran commented on SPARK-22240: thanks. Now for a question which is prob

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201407#comment-16201407 ] Hyukjin Kwon commented on SPARK-22240: -- There were multiple JIRAs for this feature.

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-11 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200485#comment-16200485 ] Steve Loughran commented on SPARK-22240: What's the link to the multiline JIRA? A

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200413#comment-16200413 ] Hyukjin Kwon commented on SPARK-22240: -- Hm, but this particular issue looks more lik

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-11 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200367#comment-16200367 ] Steve Loughran commented on SPARK-22240: Amazon EMR is amazon's own fork of Spark

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200333#comment-16200333 ] Hyukjin Kwon commented on SPARK-22240: -- This is an unfortunate limitation when {{mul