[jira] [Commented] (SPARK-12624) When schema is specified, we should give better error message if actual row length doesn't match

2016-01-29 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124528#comment-15124528
 ] 

Haidar Hadi commented on SPARK-12624:
-

I am using the scala api and I do see the same issue: when the schema does not 
match the row a generic java.lang.ArrayIndexOutOfBoundsException exception is 
raised. will this PR solve the scala/java api too ?


> When schema is specified, we should give better error message if actual row 
> length doesn't match
> 
>
> Key: SPARK-12624
> URL: https://issues.apache.org/jira/browse/SPARK-12624
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, SQL
>Reporter: Reynold Xin
>Assignee: Cheng Lian
>Priority: Blocker
> Fix For: 1.6.1, 2.0.0
>
>
> The following code snippet reproduces this issue:
> {code}
> from pyspark.sql.types import StructType, StructField, IntegerType, StringType
> from pyspark.sql.types import Row
> schema = StructType([StructField("a", IntegerType()), StructField("b", 
> StringType())])
> rdd = sc.parallelize(range(10)).map(lambda x: Row(a=x))
> df = sqlContext.createDataFrame(rdd, schema)
> df.show()
> {code}
> An unintuitive {{ArrayIndexOutOfBoundsException}} exception is thrown in this 
> case:
> {code}
> ...
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at 
> org.apache.spark.sql.catalyst.expressions.GenericInternalRow.genericGet(rows.scala:227)
> at 
> org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getAs(rows.scala:35)
> at 
> org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.isNullAt(rows.scala:36)
> ...
> {code}
> We should give a better error message here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12970) Error in documentation on creating rows with schemas defined by structs

2016-01-25 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114539#comment-15114539
 ] 

Haidar Hadi edited comment on SPARK-12970 at 1/25/16 7:11 PM:
--

sure [~joshrosen] I understand. 


was (Author: hhadi):
sure [~jrose] I understand. 

> Error in documentation on creating rows with schemas defined by structs
> ---
>
> Key: SPARK-12970
> URL: https://issues.apache.org/jira/browse/SPARK-12970
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.6.0
>Reporter: Haidar Hadi
>Priority: Minor
>  Labels: documentation
>
> The provided example in this doc 
> https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
>  for creating Row from Struct is wrong
>  // Create a Row with the schema defined by struct
>  val row = Row(Row(1, 2, true))
>  // row: Row = {@link 1,2,true}
>  
> the above example does not create a Row object with schema.
> this error is in the scala docs too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12970) Error in documentation

2016-01-24 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114485#comment-15114485
 ] 

Haidar Hadi commented on SPARK-12970:
-

let's consider the following code:

import org.apache.spark.sql.types._
 val struct = StructType(StructField("f1", StringType, true) :: Nil)
 val row = Row(1)
 println(row.fieldIndex("f1"))
 
which generates the following error when executed:
Exception in thread "main" java.lang.UnsupportedOperationException: fieldIndex 
on a Row without schema is undefined.


> Error in documentation 
> ---
>
> Key: SPARK-12970
> URL: https://issues.apache.org/jira/browse/SPARK-12970
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.6.0
>Reporter: Haidar Hadi
>Priority: Minor
>  Labels: documentation
>
> The provided example in this doc 
> https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
>  for creating Row from Struct is wrong
>  // Create a Row with the schema defined by struct
>  val row = Row(Row(1, 2, true))
>  // row: Row = {@link 1,2,true}
>  
> the above example does not create a Row object with schema.
> this error is in the scala docs too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12970) Error in documentation on creating rows with schemas defined by structs

2016-01-24 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114539#comment-15114539
 ] 

Haidar Hadi commented on SPARK-12970:
-

sure [~jrose] I understand. 

> Error in documentation on creating rows with schemas defined by structs
> ---
>
> Key: SPARK-12970
> URL: https://issues.apache.org/jira/browse/SPARK-12970
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.6.0
>Reporter: Haidar Hadi
>Priority: Minor
>  Labels: documentation
>
> The provided example in this doc 
> https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
>  for creating Row from Struct is wrong
>  // Create a Row with the schema defined by struct
>  val row = Row(Row(1, 2, true))
>  // row: Row = {@link 1,2,true}
>  
> the above example does not create a Row object with schema.
> this error is in the scala docs too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12970) Error in documentation on creating rows with schemas defined by structs

2016-01-24 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114485#comment-15114485
 ] 

Haidar Hadi edited comment on SPARK-12970 at 1/24/16 9:09 PM:
--

[~srowen]
let's consider the following code:

import org.apache.spark.sql.types._
 val struct = StructType(StructField("f1", StringType, true) :: Nil)
 val row = Row(1)
 println(row.fieldIndex("f1"))
 
which generates the following error when executed:
Exception in thread "main" java.lang.UnsupportedOperationException: fieldIndex 
on a Row without schema is undefined.

Therefore, I do not think it is taking the struct schema as param in 
constructing the Row object. 


was (Author: hhadi):
let's consider the following code:

import org.apache.spark.sql.types._
 val struct = StructType(StructField("f1", StringType, true) :: Nil)
 val row = Row(1)
 println(row.fieldIndex("f1"))
 
which generates the following error when executed:
Exception in thread "main" java.lang.UnsupportedOperationException: fieldIndex 
on a Row without schema is undefined.


> Error in documentation on creating rows with schemas defined by structs
> ---
>
> Key: SPARK-12970
> URL: https://issues.apache.org/jira/browse/SPARK-12970
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.6.0
>Reporter: Haidar Hadi
>Priority: Minor
>  Labels: documentation
>
> The provided example in this doc 
> https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
>  for creating Row from Struct is wrong
>  // Create a Row with the schema defined by struct
>  val row = Row(Row(1, 2, true))
>  // row: Row = {@link 1,2,true}
>  
> the above example does not create a Row object with schema.
> this error is in the scala docs too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-12970) Error in documentation

2016-01-23 Thread Haidar Hadi (JIRA) 
Haidar Hadi created SPARK-12970:
---

 Summary: Error in documentation 
 Key: SPARK-12970
 URL: https://issues.apache.org/jira/browse/SPARK-12970
 Project: Spark
  Issue Type: Bug
  Components: Documentation
Affects Versions: 1.6.0
Reporter: Haidar Hadi
Priority: Minor


The provided example in this doc 
https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
 for creating Row from Struct is wrong
 // Create a Row with the schema defined by struct
 val row = Row(Row(1, 2, true))
 // row: Row = {@link 1,2,true}
 
the above example does not create a Row object with schema.
this error is in the scala docs too. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-12970) Error in documentation

2016-01-23 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114060#comment-15114060
 ] 

Haidar Hadi edited comment on SPARK-12970 at 1/24/16 12:25 AM:
---

I do not mind, but I really do not know how to construct a Row with Schema.
Haidar. 



was (Author: hhadi):
I do not mind, but I really do not know how to :(
Haidar. 


> Error in documentation 
> ---
>
> Key: SPARK-12970
> URL: https://issues.apache.org/jira/browse/SPARK-12970
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.6.0
>Reporter: Haidar Hadi
>Priority: Minor
>  Labels: documentation
>
> The provided example in this doc 
> https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
>  for creating Row from Struct is wrong
>  // Create a Row with the schema defined by struct
>  val row = Row(Row(1, 2, true))
>  // row: Row = {@link 1,2,true}
>  
> the above example does not create a Row object with schema.
> this error is in the scala docs too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-12970) Error in documentation

2016-01-23 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114060#comment-15114060
 ] 

Haidar Hadi commented on SPARK-12970:
-

I do not mind, but I really do not know how to :(
Haidar. 


> Error in documentation 
> ---
>
> Key: SPARK-12970
> URL: https://issues.apache.org/jira/browse/SPARK-12970
> Project: Spark
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.6.0
>Reporter: Haidar Hadi
>Priority: Minor
>  Labels: documentation
>
> The provided example in this doc 
> https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/types/StructType.html
>  for creating Row from Struct is wrong
>  // Create a Row with the schema defined by struct
>  val row = Row(Row(1, 2, true))
>  // row: Row = {@link 1,2,true}
>  
> the above example does not create a Row object with schema.
> this error is in the scala docs too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-1902) Spark shell prints error when :4040 port already in use

2015-07-14 Thread Haidar Hadi (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626762#comment-14626762
 ] 

Haidar Hadi commented on SPARK-1902:


I am still seeing this error , I have spark running on Mesos. Any idea how to 
fix it ?
15/07/14 17:41:05 WARN AbstractLifeCycle: FAILED 
SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in 
use

java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:437)
at sun.nio.ch.Net.bind(Net.java:429)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at 
org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at 
org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at 
org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at 
org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at 
org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at 
org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:228)
at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
at org.apache.spark.ui.JettyUtils$$anonfun$2.apply(JettyUtils.scala:238)
at 
org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982)
at 
org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:238)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:117)
at 
org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448)
at 
org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:448)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.init(SparkContext.scala:448)
at com.myclass.MyClassTest$class.build_context(MyClassTes.scala:233)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/07/14 17:41:05 WARN AbstractLifeCycle: FAILED 
org.spark-project.jetty.server.Server@35fe2125: java.net.BindException: Address 
already in use

Here is the line of code that is causing it :  val sparkContext = new 
org.apache.spark.SparkContext(conf)



 Spark shell prints error when :4040 port already in use
 ---

 Key: SPARK-1902
 URL: https://issues.apache.org/jira/browse/SPARK-1902
 Project: Spark
  Issue Type: Bug
  Components: Spark Core
Affects Versions: 1.0.0
Reporter: Andrew Ash
Assignee: Andrew Ash
 Fix For: 1.1.0


 When running two shells on the same machine, I get the below error.  The 
 issue is that the first shell takes port 4040, then the next tries tries 4040 
 and fails so falls back to 4041, then a third would try 4040 and 4041 before 
 landing on 4042, etc.
 We should catch the error and instead log as Unable to use port 4041; 
 already in use.  Attempting port 4042...
 {noformat}
 14/05/22 11:31:54 WARN component.AbstractLifeCycle: FAILED 
 SelectChannelConnector@0.0.0.0:4041: java.net.BindException: Address already 
 in use
 java.net.BindException: Address already in use
 at sun.nio.ch.Net.bind0(Native Method)
 at sun.nio.ch.Net.bind(Net.java:444)
 at sun.nio.ch.Net.bind(Net.java:436)
 at 
 sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
 at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
 at 
 org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
 at 
 org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
 at