[jira] [Commented] (SPARK-8288) ScalaReflection should also try apply methods defined in companion objects when inferring schema from a Product type
[
https://issues.apache.org/jira/browse/SPARK-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181679#comment-16181679
]
Jithin Thomas commented on SPARK-8288:
--
Hi, is there a temporary fix for this issue?
I've also been trying to get Spark SQL to infer the schema of my
Scrooge-generated scala classes.
Thanks,
Jithin
> ScalaReflection should also try apply methods defined in companion objects
> when inferring schema from a Product type
>
>
> Key: SPARK-8288
> URL: https://issues.apache.org/jira/browse/SPARK-8288
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.4.0
>Reporter: Cheng Lian
>
> This ticket is derived from PARQUET-293 (which actually describes a Spark SQL
> issue).
> My comment on that issue quoted below:
> {quote}
> ... The reason of this exception is that, the Scala code Scrooge generates
> is actually a trait extending {{Product}}:
> {code}
> trait Junk
> extends ThriftStruct
> with scala.Product2[Long, String]
> with java.io.Serializable
> {code}
> while Spark expects a case class, something like:
> {code}
> case class Junk(junkID: Long, junkString: String)
> {code}
> The key difference here is that the latter case class version has a
> constructor whose arguments can be transformed into fields of the DataFrame
> schema. The exception was thrown because Spark can't find such a constructor
> from trait {{Junk}}.
> {quote}
> We can make {{ScalaReflection}} try {{apply}} methods in companion objects,
> so that trait types generated by Scrooge can also be used for Spark SQL
> schema inference.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19061) Master Web UI does not link to correct Application UI in standalone cluster mode
[
https://issues.apache.org/jira/browse/SPARK-19061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795323#comment-15795323
]
Thomas commented on SPARK-19061:
I commented on the existing issue, but I don't have the rights to reopen it.
[~srowen], would you mind reopening it for me?
> Master Web UI does not link to correct Application UI in standalone cluster
> mode
>
>
> Key: SPARK-19061
> URL: https://issues.apache.org/jira/browse/SPARK-19061
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.1.0
>Reporter: Thomas
>
> I have the same issue as in SPARK-11782. Here are the steps to reproduce it:
> * On node1, launch master:
> {code}
> ./bin/start-master.sh
> {code}
> The Master UI is available at http://node1:8080 and shows spark://node1:7077
> as master URL and spark://node1:6066 as master REST URL.
> * On node2, launch worker:
> {code}
> ./bin/start-slave.sh spark://node1:7077
> {code}
> * Upload myapp.jar to both nodes.
> * On *node1*, submit a job in standalone cluster mode:
> {code}
> ./bin/spark-submit --master spark://node1:6066 --deploy-mode cluster
> --supervise --class myMainClass --total-executor-cores 1 myapp.jar
> {code}
> What I get is:
> * A driver associated with my job, running on node2 (as expected in cluster
> mode).
> * The Application UI accessible on http://node2:4040
> The two following links are wrong:
> * On the Master UI, under "Running Application", column "Name" link on
> "myMainClass" points to http://node1:4040
> * On the Master UI, under "Running Application", column "Application ID", on
> the page of my application ID (http://node1:8080/app/?appId=myAppId), the
> "Application Detail UI" link also points to http://node1:4040
> The problem is that http://node1:4040 does not lead anywhere. The expected
> behaviour would be to replace those two links by http://node2:4040 (which is
> where the Application UI actually is).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11782) Master Web UI should link to correct Application UI in cluster mode
[
https://issues.apache.org/jira/browse/SPARK-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795322#comment-15795322
]
Thomas commented on SPARK-11782:
Did you manage to solve this, [~j4nu5]?
I have the same issue as described, here, with Spark 2.1.0.
Here are the detailed steps to reproduce it:
* On node1, launch master:
{code}
./bin/start-master.sh
{code}
The Master UI is available at http://node1:8080 and shows spark://node1:7077 as
master URL and spark://node1:6066 as master REST URL.
* On node2, launch worker:
{code}
./bin/start-slave.sh spark://node1:7077
{code}
* Upload myapp.jar to both nodes.
* On *node1*, submit a job in standalone cluster mode:
{code}
./bin/spark-submit --master spark://node1:6066 --deploy-mode cluster
--supervise --class myMainClass --total-executor-cores 1 myapp.jar
{code}
What I get is:
* A driver associated with my job, running on node2 (as expected in cluster
mode).
* The Application UI accessible on http://node2:4040
The two following links are wrong:
* On the Master UI, under "Running Application", column "Name" link on
"myMainClass" points to http://node1:4040
* On the Master UI, under "Running Application", column "Application ID", on
the page of my application ID (http://node1:8080/app/?appId=myAppId), the
"Application Detail UI" link also points to http://node1:4040
The problem is that http://node1:4040 does not lead anywhere. The expected
behaviour would be to replace those two links by http://node2:4040 (which is
where the Application UI actually is).
> Master Web UI should link to correct Application UI in cluster mode
> ---
>
> Key: SPARK-11782
> URL: https://issues.apache.org/jira/browse/SPARK-11782
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 1.4.1
>Reporter: Matthias Niehoff
>Priority: Minor
>
> - Running a standalone cluster, with node1 as master
> - Submit an application to cluster with deploy-mode=cluster
> - Application driver is on node other than node1 (i.e. node3)
> => master WebUI links to node1:4040 for Application Detail UI and not to
> node3:4040
> As the master knows on which worker the driver is running, it should be
> possible to show the correct link to the Application Detail UI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19061) Master Web UI does not link to correct Application UI in standalone cluster mode
Thomas created SPARK-19061:
--
Summary: Master Web UI does not link to correct Application UI in
standalone cluster mode
Key: SPARK-19061
URL: https://issues.apache.org/jira/browse/SPARK-19061
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 2.1.0
Reporter: Thomas
I have the same issue as in SPARK-11782. Here are the steps to reproduce it:
* On node1, launch master:
{code}
./bin/start-master.sh
{code}
The Master UI is available at http://node1:8080 and shows spark://node1:7077 as
master URL and spark://node1:6066 as master REST URL.
* On node2, launch worker:
{code}
./bin/start-slave.sh spark://node1:7077
{code}
* Upload myapp.jar to both nodes.
* On *node1*, submit a job in standalone cluster mode:
{code}
./bin/spark-submit --master spark://node1:6066 --deploy-mode cluster
--supervise --class myMainClass --total-executor-cores 1 myapp.jar
{code}
What I get is:
* A driver associated with my job, running on node2 (as expected in cluster
mode).
* The Application UI accessible on http://node2:4040
The two following links are wrong:
* On the Master UI, under "Running Application", column "Name" link on
"myMainClass" points to http://node1:4040
* On the Master UI, under "Running Application", column "Application ID", on
the page of my application ID (http://node1:8080/app/?appId=myAppId), the
"Application Detail UI" link also points to http://node1:4040
The problem is that http://node1:4040 does not lead anywhere. The expected
behaviour would be to replace those two links by http://node2:4040 (which is
where the Application UI actually is).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-16114) Add network word count example
James Thomas created SPARK-16114: Summary: Add network word count example Key: SPARK-16114 URL: https://issues.apache.org/jira/browse/SPARK-16114 Project: Spark Issue Type: Sub-task Reporter: James Thomas -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12180) DataFrame.join() in PySpark gives misleading exception when column name exists on both side
[
https://issues.apache.org/jira/browse/SPARK-12180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059816#comment-15059816
]
Daniel Thomas commented on SPARK-12180:
---
Here is the code. Without renaming the columns it was throwing the exception.
{code}
sel_starts = starts.select('uuid', 'at').withColumnRenamed('uuid',
'uuid_x').withColumnRenamed('at', 'at_x')
sel_closes = closes.select('uuid', 'at', 'session_uuid', 'total_session_sec')
start_close = sel_starts.join(sel_closes, sel_starts['uuid_x'] ==
sel_closes['session_uuid'])
start_close.cache()
start_close.take(1)
{code}
> DataFrame.join() in PySpark gives misleading exception when column name
> exists on both side
> ---
>
> Key: SPARK-12180
> URL: https://issues.apache.org/jira/browse/SPARK-12180
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.5.2
>Reporter: Daniel Thomas
>
> When joining two DataFrames on a column 'session_uuid' I got the following
> exception, because both DataFrames hat a column called 'at'. The exception is
> misleading in the cause and in the column causing the problem. Renaming the
> column fixed the exception.
> ---
> Py4JJavaError Traceback (most recent call last)
> /Applications/spark-1.5.2-bin-hadoop2.4/python/pyspark/sql/utils.py in
> deco(*a, **kw)
> 35 try:
> ---> 36 return f(*a, **kw)
> 37 except py4j.protocol.Py4JJavaError as e:
> /Applications/spark-1.5.2-bin-hadoop2.4/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py
> in get_return_value(answer, gateway_client, target_id, name)
> 299 'An error occurred while calling {0}{1}{2}.\n'.
> --> 300 format(target_id, '.', name), value)
> 301 else:
> Py4JJavaError: An error occurred while calling o484.join.
> : org.apache.spark.sql.AnalysisException: resolved attribute(s)
> session_uuid#3278 missing from
> uuid_x#9078,total_session_sec#9115L,at#3248,session_uuid#9114,uuid#9117,at#9084
> in operator !Join Inner, Some((uuid_x#9078 = session_uuid#3278));
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:154)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:49)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
> at
> org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:914)
> at org.apache.spark.sql.DataFrame.(DataFrame.scala:132)
> at
> org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154)
> at org.apache.spark.sql.DataFrame.join(DataFrame.scala:553)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> at py4j.Gateway.invoke(Gateway.java:259)
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:207)
> at java.lang.Thread.run(Thread.java:745)
> During handling of the above exception, another exception occurred:
> AnalysisException Traceback (most recent call last)
> in ()
> 1 sel_starts = starts.select('uuid', 'at').withColumnRenamed('uuid',
> 'uuid_x')#.withColumnRenamed('at', 'at_x')
> 2 sel_closes = closes.select('uuid', 'at', 'session_uuid',
> 'total_session_sec')
> > 3 start_close = sel_starts.join(sel_closes, sel_starts['uuid_x'] ==
> sel_closes['session_uuid'])
> 4 start_close.cache()
> 5 start_close.take(1)
> /Applications/spark-1.5.2-bin-hadoop2.4/python/pyspark/sql/dataframe.py in
> join(self, other, on, how)
> 579 on = on[0]
> 580 if how is None:
> --> 581
[jira] [Created] (SPARK-12180) DataFrame.join() in PySpark gives misleading exception when column name exists on both side
Daniel Thomas created SPARK-12180:
-
Summary: DataFrame.join() in PySpark gives misleading exception
when column name exists on both side
Key: SPARK-12180
URL: https://issues.apache.org/jira/browse/SPARK-12180
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.5.2
Reporter: Daniel Thomas
When joining two DataFrames on a column 'session_uuid' I got the following
exception, because both DataFrames hat a column called 'at'. The exception is
misleading in the cause and in the column causing the problem. Renaming the
column fixed the exception.
---
Py4JJavaError Traceback (most recent call last)
/Applications/spark-1.5.2-bin-hadoop2.4/python/pyspark/sql/utils.py in deco(*a,
**kw)
35 try:
---> 36 return f(*a, **kw)
37 except py4j.protocol.Py4JJavaError as e:
/Applications/spark-1.5.2-bin-hadoop2.4/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py
in get_return_value(answer, gateway_client, target_id, name)
299 'An error occurred while calling {0}{1}{2}.\n'.
--> 300 format(target_id, '.', name), value)
301 else:
Py4JJavaError: An error occurred while calling o484.join.
: org.apache.spark.sql.AnalysisException: resolved attribute(s)
session_uuid#3278 missing from
uuid_x#9078,total_session_sec#9115L,at#3248,session_uuid#9114,uuid#9117,at#9084
in operator !Join Inner, Some((uuid_x#9078 = session_uuid#3278));
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:154)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:49)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
at
org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:914)
at org.apache.spark.sql.DataFrame.(DataFrame.scala:132)
at
org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154)
at org.apache.spark.sql.DataFrame.join(DataFrame.scala:553)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
During handling of the above exception, another exception occurred:
AnalysisException Traceback (most recent call last)
in ()
1 sel_starts = starts.select('uuid', 'at').withColumnRenamed('uuid',
'uuid_x')#.withColumnRenamed('at', 'at_x')
2 sel_closes = closes.select('uuid', 'at', 'session_uuid',
'total_session_sec')
> 3 start_close = sel_starts.join(sel_closes, sel_starts['uuid_x'] ==
sel_closes['session_uuid'])
4 start_close.cache()
5 start_close.take(1)
/Applications/spark-1.5.2-bin-hadoop2.4/python/pyspark/sql/dataframe.py in
join(self, other, on, how)
579 on = on[0]
580 if how is None:
--> 581 jdf = self._jdf.join(other._jdf, on._jc, "inner")
582 else:
583 assert isinstance(how, basestring), "how should be
basestring"
/Applications/spark-1.5.2-bin-hadoop2.4/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py
in __call__(self, *args)
536 answer = self.gateway_client.send_command(command)
537 return_value = get_return_value(answer, self.gateway_client,
--> 538 self.target_id, self.name)
539
540 for temp_arg in temp_args:
/Applications/spark-1.5.2-bin-hadoop2.4/python/pyspark/sql/utils.py in deco(*a,
**kw)
38 s = e.java_exception.toString()
39 if
[jira] [Commented] (SPARK-10262) Add @Since annotation to ml.attribute
[ https://issues.apache.org/jira/browse/SPARK-10262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901359#comment-14901359 ] Tijo Thomas commented on SPARK-10262: - I could not complete this as I got stuck up with my other tasks. I appologize for the delay caused. Yashwanth Kumar could you handle this issue. > Add @Since annotation to ml.attribute > - > > Key: SPARK-10262 > URL: https://issues.apache.org/jira/browse/SPARK-10262 > Project: Spark > Issue Type: Sub-task > Components: Documentation, ML >Reporter: Xiangrui Meng >Priority: Minor > Labels: starter > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10262) Add @Since annotation to ml.attribute
[ https://issues.apache.org/jira/browse/SPARK-10262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805079#comment-14805079 ] Tijo Thomas commented on SPARK-10262: - I am sorry for the delay . There are many file and i am almost done with this. I will commit all the changes by Sat EOD . > Add @Since annotation to ml.attribute > - > > Key: SPARK-10262 > URL: https://issues.apache.org/jira/browse/SPARK-10262 > Project: Spark > Issue Type: Sub-task > Components: Documentation, ML >Reporter: Xiangrui Meng >Priority: Minor > Labels: starter > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10261) Add @Since annotation to ml.evaluation
[ https://issues.apache.org/jira/browse/SPARK-10261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724975#comment-14724975 ] Tijo Thomas commented on SPARK-10261: - I am working on this issue. > Add @Since annotation to ml.evaluation > -- > > Key: SPARK-10261 > URL: https://issues.apache.org/jira/browse/SPARK-10261 > Project: Spark > Issue Type: Sub-task > Components: Documentation, ML >Reporter: Xiangrui Meng >Priority: Minor > Labels: starter > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10262) Add @Since annotation to ml.attribute
[ https://issues.apache.org/jira/browse/SPARK-10262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14725388#comment-14725388 ] Tijo Thomas commented on SPARK-10262: - I am working on this. > Add @Since annotation to ml.attribute > - > > Key: SPARK-10262 > URL: https://issues.apache.org/jira/browse/SPARK-10262 > Project: Spark > Issue Type: Sub-task > Components: Documentation, ML >Reporter: Xiangrui Meng >Priority: Minor > Labels: starter > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-10375) Setting the driver memory with SparkConf().set("spark.driver.memory","1g") does not work
Thomas created SPARK-10375:
--
Summary: Setting the driver memory with
SparkConf().set("spark.driver.memory","1g") does not work
Key: SPARK-10375
URL: https://issues.apache.org/jira/browse/SPARK-10375
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.3.0
Environment: Running with yarn
Reporter: Thomas
Priority: Minor
When running pyspark 1.3.0 with yarn, the following code has no effect:
pyspark.SparkConf().set("spark.driver.memory","1g")
The Environment tab in yarn shows that the driver has 1g, however, the
Executors tab only shows 512 M (the default value) for the driver memory. This
issue goes away when the driver memory is specified via the command line (i.e.
--driver-memory 1g)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10264) Add @Since annotation to ml.recoomendation
[ https://issues.apache.org/jira/browse/SPARK-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14721682#comment-14721682 ] Tijo Thomas commented on SPARK-10264: - I am working on this. Thanks Add @Since annotation to ml.recoomendation -- Key: SPARK-10264 URL: https://issues.apache.org/jira/browse/SPARK-10264 Project: Spark Issue Type: Sub-task Components: Documentation, ML Reporter: Xiangrui Meng Priority: Minor Labels: starter -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-10113) Support for unsigned Parquet logical types
Jordan Thomas created SPARK-10113:
-
Summary: Support for unsigned Parquet logical types
Key: SPARK-10113
URL: https://issues.apache.org/jira/browse/SPARK-10113
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 1.5.0
Reporter: Jordan Thomas
Add support for unsigned Parquet logical types UINT_16, UINT_32 and UINT_64.
{code}
org.apache.spark.sql.AnalysisException: Illegal Parquet type: INT64 (UINT_64);
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.illegalType$1(CatalystSchemaConverter.scala:130)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.convertPrimitiveField(CatalystSchemaConverter.scala:169)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.convertField(CatalystSchemaConverter.scala:115)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter$$anonfun$2.apply(CatalystSchemaConverter.scala:97)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter$$anonfun$2.apply(CatalystSchemaConverter.scala:94)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.org$apache$spark$sql$parquet$CatalystSchemaConverter$$convert(CatalystSchemaConverter.scala:94)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter$$anonfun$convertGroupField$1.apply(CatalystSchemaConverter.scala:200)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter$$anonfun$convertGroupField$1.apply(CatalystSchemaConverter.scala:200)
at scala.Option.fold(Option.scala:158)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.convertGroupField(CatalystSchemaConverter.scala:200)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.convertField(CatalystSchemaConverter.scala:116)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter$$anonfun$2.apply(CatalystSchemaConverter.scala:97)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter$$anonfun$2.apply(CatalystSchemaConverter.scala:94)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.org$apache$spark$sql$parquet$CatalystSchemaConverter$$convert(CatalystSchemaConverter.scala:94)
at
org.apache.spark.sql.parquet.CatalystSchemaConverter.convert(CatalystSchemaConverter.scala:91)
at
org.apache.spark.sql.parquet.ParquetRelation$$anonfun$readSchemaFromFooter$2.apply(ParquetRelation.scala:734)
at
org.apache.spark.sql.parquet.ParquetRelation$$anonfun$readSchemaFromFooter$2.apply(ParquetRelation.scala:734)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.parquet.ParquetRelation$.readSchemaFromFooter(ParquetRelation.scala:734)
at
org.apache.spark.sql.parquet.ParquetRelation$$anonfun$28$$anonfun$apply$8.apply(ParquetRelation.scala:714)
at
org.apache.spark.sql.parquet.ParquetRelation$$anonfun$28$$anonfun$apply$8.apply(ParquetRelation.scala:713)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
[jira] [Created] (SPARK-8886) Python style usually don't add space before/after the = in named parameters
Tijo Thomas created SPARK-8886: -- Summary: Python style usually don't add space before/after the = in named parameters Key: SPARK-8886 URL: https://issues.apache.org/jira/browse/SPARK-8886 Project: Spark Issue Type: Sub-task Reporter: Tijo Thomas Priority: Trivial python style usually don't add space before/after the = in named parameters. can you submit a follow up patch to fix that? thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8732) Compilation warning for existentials code
Tijo Thomas created SPARK-8732: -- Summary: Compilation warning for existentials code Key: SPARK-8732 URL: https://issues.apache.org/jira/browse/SPARK-8732 Project: Spark Issue Type: Improvement Components: Build Reporter: Tijo Thomas Priority: Trivial Compilation warning for Scala code for using existential 1. RBackendHandler.scala 2. CatalystTypeConverters.scala Need to add import. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7399) Master fails on 2.11 with compilation error
[
https://issues.apache.org/jira/browse/SPARK-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543124#comment-14543124
]
Tijo Thomas commented on SPARK-7399:
Jira Resolution status changed from Fixed to Pending Close . I think this is
due to the bug in JIRA. Could you update the status.
Master fails on 2.11 with compilation error
---
Key: SPARK-7399
URL: https://issues.apache.org/jira/browse/SPARK-7399
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.4.0
Reporter: Iulian Dragos
Assignee: Tijo Thomas
Priority: Blocker
Fix For: 1.4.0
The current code in master (and 1.4 branch) fails on 2.11 with the following
compilation error:
{code}
[error] /home/ubuntu/workspace/Apache Spark (master) on
2.11/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala:78: in
object RDDOperationScope, multiple overloaded alternatives of method
withScope define default arguments.
[error] private[spark] object RDDOperationScope {
[error] ^
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7399) Master fails on 2.11 with compilation error
[
https://issues.apache.org/jira/browse/SPARK-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532062#comment-14532062
]
Tijo Thomas commented on SPARK-7399:
Raised a pull request https://github.com/apache/spark/pull/5966
Master fails on 2.11 with compilation error
---
Key: SPARK-7399
URL: https://issues.apache.org/jira/browse/SPARK-7399
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.4.0
Reporter: Iulian Dragos
The current code in master (and 1.4 branch) fails on 2.11 with the following
compilation error:
{code}
[error] /home/ubuntu/workspace/Apache Spark (master) on
2.11/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala:78: in
object RDDOperationScope, multiple overloaded alternatives of method
withScope define default arguments.
[error] private[spark] object RDDOperationScope {
[error] ^
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-7399) Master fails on 2.11 with compilation error
[
https://issues.apache.org/jira/browse/SPARK-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tijo Thomas updated SPARK-7399:
---
Comment: was deleted
(was: Raised a pull request https://github.com/apache/spark/pull/5966)
Master fails on 2.11 with compilation error
---
Key: SPARK-7399
URL: https://issues.apache.org/jira/browse/SPARK-7399
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.4.0
Reporter: Iulian Dragos
The current code in master (and 1.4 branch) fails on 2.11 with the following
compilation error:
{code}
[error] /home/ubuntu/workspace/Apache Spark (master) on
2.11/core/src/main/scala/org/apache/spark/rdd/RDDOperationScope.scala:78: in
object RDDOperationScope, multiple overloaded alternatives of method
withScope define default arguments.
[error] private[spark] object RDDOperationScope {
[error] ^
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-7087) Scala Version Change script is dependent on current working directory
Tijo Thomas created SPARK-7087: -- Summary: Scala Version Change script is dependent on current working directory Key: SPARK-7087 URL: https://issues.apache.org/jira/browse/SPARK-7087 Project: Spark Issue Type: Bug Components: Build Reporter: Tijo Thomas Priority: Minor script change-version-to-2.10.sh and change-version-to-2.11.sh is not working if the current working directory is not spark for example if I run the script inside dev folder I get the following error /spark/dev$ ./change-version-to-2.10.sh sed: can't read pom.xml: No such file or directory I was trying to configure Jenkins to build using scala 2.10 and 2.11 one after another. In my jenkins I need to use absolute path in pre build step Find command is searching for all .pom file assuming that the current working directory is spark -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6928) spark-shell stops working after the replay command
[ https://issues.apache.org/jira/browse/SPARK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506930#comment-14506930 ] Tijo Thomas commented on SPARK-6928: You may close this jira issue as this is not so important spark-shell stops working after the replay command -- Key: SPARK-6928 URL: https://issues.apache.org/jira/browse/SPARK-6928 Project: Spark Issue Type: Bug Components: Spark Shell Affects Versions: 1.3.0 Environment: Scala Version :Scala-2.10 Reporter: Tijo Thomas Step to reproduce this issues. Step 1 : scala sc.parallelize(1 to 10).map(_+2).count(); res0: Long = 10 Step 2 : scala :replay Replaying: sc.parallelize(1 to 10).map(_+2).count(); console:8: error: not found: value sc sc.parallelize(1 to 10).map(_+2).count(); ^ // Note : After Replay command , Non of the spark api's are working as the SparkContext has gone out of scope. eg: getting this exception as given below scala exit error: while compiling: console during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: last tree to typer: Apply(constructor $read) symbol: constructor $read in class $read (flags: method triedcooking) symbol definition: def init(): $line20.$read tpe: $line20.$read symbol owners: constructor $read - class $read - package $line20 context owners: class iwC - package $line20 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-6928) spark-shell stops working after the replay command
[ https://issues.apache.org/jira/browse/SPARK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tijo Thomas closed SPARK-6928. -- Resolution: Not A Problem spark-shell stops working after the replay command -- Key: SPARK-6928 URL: https://issues.apache.org/jira/browse/SPARK-6928 Project: Spark Issue Type: Bug Components: Spark Shell Affects Versions: 1.3.0 Environment: Scala Version :Scala-2.10 Reporter: Tijo Thomas Step to reproduce this issues. Step 1 : scala sc.parallelize(1 to 10).map(_+2).count(); res0: Long = 10 Step 2 : scala :replay Replaying: sc.parallelize(1 to 10).map(_+2).count(); console:8: error: not found: value sc sc.parallelize(1 to 10).map(_+2).count(); ^ // Note : After Replay command , Non of the spark api's are working as the SparkContext has gone out of scope. eg: getting this exception as given below scala exit error: while compiling: console during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: last tree to typer: Apply(constructor $read) symbol: constructor $read in class $read (flags: method triedcooking) symbol definition: def init(): $line20.$read tpe: $line20.$read symbol owners: constructor $read - class $read - package $line20 context owners: class iwC - package $line20 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6928) spark-shell stops working after the replay command
Tijo Thomas created SPARK-6928: -- Summary: spark-shell stops working after the replay command Key: SPARK-6928 URL: https://issues.apache.org/jira/browse/SPARK-6928 Project: Spark Issue Type: Bug Components: Spark Shell Affects Versions: 1.3.0 Environment: Scala Version :Scala-2.10 Reporter: Tijo Thomas Step to reproduce this issues. Step 1 : scala sc.parallelize(1 to 10).map(_+2).count(); res0: Long = 10 Step 2 : scala :replay Replaying: sc.parallelize(1 to 10).map(_+2).count(); console:8: error: not found: value sc sc.parallelize(1 to 10).map(_+2).count(); ^ // Note : After Replay command , Non of the spark api's are working as the SparkContext has gone out of scope. eg: getting this exception as given below scala exit error: while compiling: console during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: last tree to typer: Apply(constructor $read) symbol: constructor $read in class $read (flags: method triedcooking) symbol definition: def init(): $line20.$read tpe: $line20.$read symbol owners: constructor $read - class $read - package $line20 context owners: class iwC - package $line20 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6928) spark-shell stops working after the replay command
[ https://issues.apache.org/jira/browse/SPARK-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495934#comment-14495934 ] Tijo Thomas commented on SPARK-6928: Spark shell is extending the scala shell and sc is getting lost when there is reset operation in spark REPL code. I think that need to be handled in spark REPL itself. spark-shell stops working after the replay command -- Key: SPARK-6928 URL: https://issues.apache.org/jira/browse/SPARK-6928 Project: Spark Issue Type: Bug Components: Spark Shell Affects Versions: 1.3.0 Environment: Scala Version :Scala-2.10 Reporter: Tijo Thomas Step to reproduce this issues. Step 1 : scala sc.parallelize(1 to 10).map(_+2).count(); res0: Long = 10 Step 2 : scala :replay Replaying: sc.parallelize(1 to 10).map(_+2).count(); console:8: error: not found: value sc sc.parallelize(1 to 10).map(_+2).count(); ^ // Note : After Replay command , Non of the spark api's are working as the SparkContext has gone out of scope. eg: getting this exception as given below scala exit error: while compiling: console during phase: jvm library version: version 2.10.4 compiler version: version 2.10.4 reconstructed args: last tree to typer: Apply(constructor $read) symbol: constructor $read in class $read (flags: method triedcooking) symbol definition: def init(): $line20.$read tpe: $line20.$read symbol owners: constructor $read - class $read - package $line20 context owners: class iwC - package $line20 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6767) Documentation error in Spark SQL Readme file
Tijo Thomas created SPARK-6767:
--
Summary: Documentation error in Spark SQL Readme file
Key: SPARK-6767
URL: https://issues.apache.org/jira/browse/SPARK-6767
Project: Spark
Issue Type: Bug
Components: Documentation, SQL
Affects Versions: 1.3.0
Reporter: Tijo Thomas
Priority: Trivial
Error in Spark SQL Documentation file . The sample script for SQL DSL
throwing below error
scala query.where('key 30).select(avg('key)).collect()
console:43: error: value is not a member of Symbol
query.where('key 30).select(avg('key)).collect()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6767) Documentation error in Spark SQL Readme file
[
https://issues.apache.org/jira/browse/SPARK-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486650#comment-14486650
]
Tijo Thomas commented on SPARK-6767:
Could you please change the status of this issue and assign this to me ?
Documentation error in Spark SQL Readme file
Key: SPARK-6767
URL: https://issues.apache.org/jira/browse/SPARK-6767
Project: Spark
Issue Type: Bug
Components: Documentation, SQL
Affects Versions: 1.3.0
Reporter: Tijo Thomas
Priority: Trivial
Error in Spark SQL Documentation file . The sample script for SQL DSL
throwing below error
scala query.where('key 30).select(avg('key)).collect()
console:43: error: value is not a member of Symbol
query.where('key 30).select(avg('key)).collect()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6739) Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types
Tijo Thomas created SPARK-6739: -- Summary: Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types Key: SPARK-6739 URL: https://issues.apache.org/jira/browse/SPARK-6739 Project: Spark Issue Type: Bug Components: Documentation, SQL Affects Versions: 1.3.0 Reporter: Tijo Thomas Priority: Trivial Missing import in example script under the section Programmatically Specifying the Schema scala val schema = | StructType( | schemaString.split( ).map(fieldName = StructField(fieldName, StringType, true))) console:25: error: not found: value StructType StructType( ^ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6739) Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types
[ https://issues.apache.org/jira/browse/SPARK-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483034#comment-14483034 ] Tijo Thomas commented on SPARK-6739: Please close this duplicate issue Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types Key: SPARK-6739 URL: https://issues.apache.org/jira/browse/SPARK-6739 Project: Spark Issue Type: Bug Components: Documentation, SQL Affects Versions: 1.3.0 Reporter: Tijo Thomas Priority: Trivial Missing import in example script under the section Programmatically Specifying the Schema scala val schema = | StructType( | schemaString.split( ).map(fieldName = StructField(fieldName, StringType, true))) console:25: error: not found: value StructType StructType( ^ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6739) Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types
[ https://issues.apache.org/jira/browse/SPARK-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483034#comment-14483034 ] Tijo Thomas edited comment on SPARK-6739 at 4/7/15 11:26 AM: - Please close this duplicate issue How ever the previous fix is not reflecting in the documentation under section Programmatically Specifying the Schema :: https://spark.apache.org/docs/latest/sql-programming-guide.html#manually-specifying-options was (Author: tijo paracka): Please close this duplicate issue How ever the previous fix is not reflecting in the documentation. Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types Key: SPARK-6739 URL: https://issues.apache.org/jira/browse/SPARK-6739 Project: Spark Issue Type: Bug Components: Documentation, SQL Affects Versions: 1.3.0 Reporter: Tijo Thomas Priority: Trivial Missing import in example script under the section Programmatically Specifying the Schema scala val schema = | StructType( | schemaString.split( ).map(fieldName = StructField(fieldName, StringType, true))) console:25: error: not found: value StructType StructType( ^ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-6739) Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types
[ https://issues.apache.org/jira/browse/SPARK-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483034#comment-14483034 ] Tijo Thomas edited comment on SPARK-6739 at 4/7/15 11:25 AM: - Please close this duplicate issue How ever the previous fix is not reflecting in the documentation. was (Author: tijo paracka): Please close this duplicate issue Spark SQL Example gives errors due to missing import of Types org.apache.spark.sql.types Key: SPARK-6739 URL: https://issues.apache.org/jira/browse/SPARK-6739 Project: Spark Issue Type: Bug Components: Documentation, SQL Affects Versions: 1.3.0 Reporter: Tijo Thomas Priority: Trivial Missing import in example script under the section Programmatically Specifying the Schema scala val schema = | StructType( | schemaString.split( ).map(fieldName = StructField(fieldName, StringType, true))) console:25: error: not found: value StructType StructType( ^ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6383) Few examples on Dataframe operation give compiler errors
[ https://issues.apache.org/jira/browse/SPARK-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377571#comment-14377571 ] Tijo Thomas commented on SPARK-6383: The Assignee: for this issues appeared as Unassigned. Could you please change the assignee to my name. Few examples on Dataframe operation give compiler errors - Key: SPARK-6383 URL: https://issues.apache.org/jira/browse/SPARK-6383 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.3.0 Reporter: Tijo Thomas Priority: Trivial Labels: DataFrame Fix For: 1.3.1, 1.4.0 The below statements give compiler errors as a) the select method doesnot accept String, Column df.select(name, df(age) + 1).show() // Need to convert String to Column b) Filtering should be based on age not on name Column df.filter(df(name) 21).show() -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6383) Few examples on Dataframe operation give compiler errors
Tijo Thomas created SPARK-6383: -- Summary: Few examples on Dataframe operation give compiler errors Key: SPARK-6383 URL: https://issues.apache.org/jira/browse/SPARK-6383 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.3.0 Reporter: Tijo Thomas Priority: Trivial The below statements give compiler errors as a) the select method doesnot accept String, Column df.select(name, df(age) + 1).show() // Need to convert String to Column b) Filtering should be based on age not on name Column df.filter(df(name) 21).show() -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
