[jira] [Resolved] (SPARK-7909) spark-ec2 and associated tools not py3 ready

[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7909. --- Resolution: Fixed spark-ec2 and associated tools not py3 ready

[jira] [Resolved] (SPARK-6289) PySpark doesn't maintain SQL date Types

[ https://issues.apache.org/jira/browse/SPARK-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-6289. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7301

[jira] [Resolved] (SPARK-7902) SQL UDF doesn't support UDT in PySpark

[ https://issues.apache.org/jira/browse/SPARK-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7902. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7301

[jira] [Updated] (SPARK-7909) spark-ec2 and associated tools not py3 ready

[ https://issues.apache.org/jira/browse/SPARK-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-7909: -- Target Version/s: 1.5.0 Priority: Blocker (was: Major) spark-ec2 and associated tools not

[jira] [Commented] (SPARK-4315) PySpark pickling of pyspark.sql.Row objects is extremely inefficient

[ https://issues.apache.org/jira/browse/SPARK-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619611#comment-14619611 ] Davies Liu commented on SPARK-4315: --- This is fixed by

[jira] [Assigned] (SPARK-4315) PySpark pickling of pyspark.sql.Row objects is extremely inefficient

[ https://issues.apache.org/jira/browse/SPARK-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-4315: - Assignee: Davies Liu PySpark pickling of pyspark.sql.Row objects is extremely inefficient

[jira] [Resolved] (SPARK-4315) PySpark pickling of pyspark.sql.Row objects is extremely inefficient

[ https://issues.apache.org/jira/browse/SPARK-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-4315. --- Resolution: Fixed Fix Version/s: 1.4.0 1.3.2 PySpark pickling of

[jira] [Commented] (SPARK-5092) Selecting from a nested structure with SparkSQL should return a nested structure

[ https://issues.apache.org/jira/browse/SPARK-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619632#comment-14619632 ] Davies Liu commented on SPARK-5092: --- cc [~marmbrus] Selecting from a nested structure

[jira] [Updated] (SPARK-8931) Fallback to interpret mode if failed to compile in codegen

[ https://issues.apache.org/jira/browse/SPARK-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8931: -- Description: And we should not fallback during testing. Fallback to interpret mode if failed to

[jira] [Closed] (SPARK-7507) pyspark.sql.types.StructType and Row should implement __iter__()

[ https://issues.apache.org/jira/browse/SPARK-7507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-7507. - Resolution: Won't Fix pyspark.sql.types.StructType and Row should implement __iter__()

[jira] [Commented] (SPARK-7507) pyspark.sql.types.StructType and Row should implement __iter__()

[ https://issues.apache.org/jira/browse/SPARK-7507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619609#comment-14619609 ] Davies Liu commented on SPARK-7507: --- For `Row`, it's similar to namedtuple, you can

[jira] [Resolved] (SPARK-8450) PySpark write.parquet raises Unsupported datatype DecimalType()

[ https://issues.apache.org/jira/browse/SPARK-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8450. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7131

[jira] [Commented] (SPARK-8408) Python OR operator is not considered while creating a column of boolean type

[ https://issues.apache.org/jira/browse/SPARK-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619622#comment-14619622 ] Davies Liu commented on SPARK-8408: --- In Python, We cannot override `or` `and` `not`, so

[jira] [Resolved] (SPARK-8408) Python OR operator is not considered while creating a column of boolean type

[ https://issues.apache.org/jira/browse/SPARK-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8408. --- Resolution: Fixed Assignee: Davies Liu Fix Version/s: 1.4.1 Python OR operator is

[jira] [Created] (SPARK-8931) Fallback to interpret mode if failed to compile in codegen

Davies Liu created SPARK-8931: - Summary: Fallback to interpret mode if failed to compile in codegen Key: SPARK-8931 URL: https://issues.apache.org/jira/browse/SPARK-8931 Project: Spark Issue

[jira] [Resolved] (SPARK-7190) UTF8String backed by binary data

[ https://issues.apache.org/jira/browse/SPARK-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7190. --- Resolution: Fixed UTF8String backed by binary data

[jira] [Resolved] (SPARK-7815) Enable UTF8String to work against memory address directly

[ https://issues.apache.org/jira/browse/SPARK-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7815. --- Resolution: Fixed Enable UTF8String to work against memory address directly

[jira] [Assigned] (SPARK-6573) Convert inbound NaN values as null

[ https://issues.apache.org/jira/browse/SPARK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-6573: - Assignee: Davies Liu Convert inbound NaN values as null --

[jira] [Resolved] (SPARK-8804) order of UTF8String is wrong if there is any non-ascii character in it

[ https://issues.apache.org/jira/browse/SPARK-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8804. --- Resolution: Fixed Fix Version/s: 1.5.0 order of UTF8String is wrong if there is any

[jira] [Created] (SPARK-8844) head/collect is broken in SparkR

Davies Liu created SPARK-8844: - Summary: head/collect is broken in SparkR Key: SPARK-8844 URL: https://issues.apache.org/jira/browse/SPARK-8844 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-8646) PySpark does not run on YARN

[ https://issues.apache.org/jira/browse/SPARK-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615293#comment-14615293 ] Davies Liu commented on SPARK-8646: --- To be clear, PySpark does NOT depends on pandas. In

[jira] [Updated] (SPARK-8745) Remove GenerateProjection

[ https://issues.apache.org/jira/browse/SPARK-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8745: -- Summary: Remove GenerateProjection (was: Remove GenerateMutableProjection) Remove GenerateProjection

[jira] [Updated] (SPARK-8745) Remove GenerateProjection

[ https://issues.apache.org/jira/browse/SPARK-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8745: -- Description: Based on discussion offline with [~marmbrus], we should remove GenerateProjection. (was:

[jira] [Commented] (SPARK-8636) CaseKeyWhen has incorrect NULL handling

[ https://issues.apache.org/jira/browse/SPARK-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614080#comment-14614080 ] Davies Liu commented on SPARK-8636: --- [~smolav] I'm just curious that how can we sort of

[jira] [Resolved] (SPARK-7401) Dot product and squared_distances should be vectorized in Vectors

[ https://issues.apache.org/jira/browse/SPARK-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7401. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 5946

[jira] [Resolved] (SPARK-8226) math function: shiftrightunsigned

[ https://issues.apache.org/jira/browse/SPARK-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8226. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7035

[jira] [Created] (SPARK-8784) Add python API for hex/unhex

Davies Liu created SPARK-8784: - Summary: Add python API for hex/unhex Key: SPARK-8784 URL: https://issues.apache.org/jira/browse/SPARK-8784 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-8632) Poor Python UDF performance because of RDD caching

[ https://issues.apache.org/jira/browse/SPARK-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14611602#comment-14611602 ] Davies Liu commented on SPARK-8632: --- [~justin.uang] Sounds interesting, could you

[jira] [Created] (SPARK-8786) Create a wrapper for BinaryType

Davies Liu created SPARK-8786: - Summary: Create a wrapper for BinaryType Key: SPARK-8786 URL: https://issues.apache.org/jira/browse/SPARK-8786 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-8786) Create a wrapper for BinaryType

[ https://issues.apache.org/jira/browse/SPARK-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8786: -- Description: The hashCode and equals() of Array[Byte] does check the bytes, we should create a wrapper

[jira] [Resolved] (SPARK-8223) math function: shiftleft

[ https://issues.apache.org/jira/browse/SPARK-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8223. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7178

[jira] [Resolved] (SPARK-8224) math function: shiftright

[ https://issues.apache.org/jira/browse/SPARK-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8224. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7178

[jira] [Resolved] (SPARK-8747) fix EqualNullSafe for binary type

[ https://issues.apache.org/jira/browse/SPARK-8747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8747. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7143

[jira] [Assigned] (SPARK-7190) UTF8String backed by binary data

[ https://issues.apache.org/jira/browse/SPARK-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-7190: - Assignee: Davies Liu UTF8String backed by binary data

[jira] [Commented] (SPARK-8745) Remove GenerateMutableProjection

[ https://issues.apache.org/jira/browse/SPARK-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612471#comment-14612471 ] Davies Liu commented on SPARK-8745: --- I can take this one, if you have not started.

[jira] [Assigned] (SPARK-8745) Remove GenerateMutableProjection

[ https://issues.apache.org/jira/browse/SPARK-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-8745: - Assignee: Davies Liu Remove GenerateMutableProjection

[jira] [Created] (SPARK-8804) order of UTF8String is wrong if there is any non-ascii character in it

Davies Liu created SPARK-8804: - Summary: order of UTF8String is wrong if there is any non-ascii character in it Key: SPARK-8804 URL: https://issues.apache.org/jira/browse/SPARK-8804 Project: Spark

[jira] [Created] (SPARK-8766) DataFrame Python API should work with column which has non-ascii character in it

Davies Liu created SPARK-8766: - Summary: DataFrame Python API should work with column which has non-ascii character in it Key: SPARK-8766 URL: https://issues.apache.org/jira/browse/SPARK-8766 Project:

[jira] [Resolved] (SPARK-8763) executing run-tests.py with Python 2.6 fails with absence of subprocess.check_output function

[ https://issues.apache.org/jira/browse/SPARK-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8763. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7161

[jira] [Resolved] (SPARK-8227) math function: unhex

[ https://issues.apache.org/jira/browse/SPARK-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8227. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7113

[jira] [Resolved] (SPARK-8766) DataFrame Python API should work with column which has non-ascii character in it

[ https://issues.apache.org/jira/browse/SPARK-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8766. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7165

[jira] [Resolved] (SPARK-8727) Add missing python api

[ https://issues.apache.org/jira/browse/SPARK-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8727. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7114

[jira] [Resolved] (SPARK-8535) PySpark : Can't create DataFrame from Pandas dataframe with no explicit column name

[ https://issues.apache.org/jira/browse/SPARK-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8535. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7124

[jira] [Commented] (SPARK-8653) Add constraint for Children expression for data type

[ https://issues.apache.org/jira/browse/SPARK-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14609579#comment-14609579 ] Davies Liu commented on SPARK-8653: --- [~rxin] With the new `ExpectsInputTypes`, we still

[jira] [Resolved] (SPARK-8723) improve code gen for divide and remainder

[ https://issues.apache.org/jira/browse/SPARK-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8723. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7111

[jira] [Resolved] (SPARK-8680) PropagateTypes is very slow when there are lots of columns

[ https://issues.apache.org/jira/browse/SPARK-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8680. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7087

[jira] [Updated] (SPARK-8680) PropagateTypes is very slow when there are lots of columns

[ https://issues.apache.org/jira/browse/SPARK-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8680: -- Assignee: Liang-Chi Hsieh PropagateTypes is very slow when there are lots of columns

[jira] [Resolved] (SPARK-8590) add code gen for ExtractValue

[ https://issues.apache.org/jira/browse/SPARK-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8590. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6982

[jira] [Resolved] (SPARK-8236) misc function: crc32

[ https://issues.apache.org/jira/browse/SPARK-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8236. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7108

[jira] [Updated] (SPARK-8450) PySpark write.parquet raises Unsupported datatype DecimalType()

[ https://issues.apache.org/jira/browse/SPARK-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8450: -- Description: I'm getting an Exception when I try to save a DataFrame with a DeciamlType as an parquet

[jira] [Resolved] (SPARK-8713) Support codegen for not thread-safe expressions

[ https://issues.apache.org/jira/browse/SPARK-8713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8713. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7101

[jira] [Created] (SPARK-8738) Generate better error message in Python for AnalysisException

Davies Liu created SPARK-8738: - Summary: Generate better error message in Python for AnalysisException Key: SPARK-8738 URL: https://issues.apache.org/jira/browse/SPARK-8738 Project: Spark

[jira] [Resolved] (SPARK-8738) Generate better error message in Python for AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-8738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8738. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7135

[jira] [Updated] (SPARK-6360) For Spark 1.1 and 1.2, after any RDD transformations, calling saveAsParquetFile over a SchemaRDD with decimal or UDT column throws

[ https://issues.apache.org/jira/browse/SPARK-6360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-6360: -- Description: Spark shell session for reproduction (use {{:paste}}): {noformat} import

[jira] [Resolved] (SPARK-8741) Remove e and pi from DataFrame functions

[ https://issues.apache.org/jira/browse/SPARK-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8741. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7137

[jira] [Assigned] (SPARK-7902) SQL UDF doesn't support UDT in PySpark

[ https://issues.apache.org/jira/browse/SPARK-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-7902: - Assignee: Davies Liu SQL UDF doesn't support UDT in PySpark

[jira] [Resolved] (SPARK-8235) misc function: sha1 / sha

[ https://issues.apache.org/jira/browse/SPARK-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8235. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6963

[jira] [Created] (SPARK-8713) Support codegen for not thread-safe expressions

Davies Liu created SPARK-8713: - Summary: Support codegen for not thread-safe expressions Key: SPARK-8713 URL: https://issues.apache.org/jira/browse/SPARK-8713 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-7810) rdd.py _load_from_socket cannot load data from jvm socket if ipv6 is used

[ https://issues.apache.org/jira/browse/SPARK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-7810. --- Resolution: Fixed Fix Version/s: 1.6.0 1.3.2 1.4.1 Issue

[jira] [Resolved] (SPARK-8579) Support arbitrary object in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-8579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8579. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6959

[jira] [Updated] (SPARK-7810) rdd.py _load_from_socket cannot load data from jvm socket if ipv6 is used

[ https://issues.apache.org/jira/browse/SPARK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-7810: -- Fix Version/s: (was: 1.4.1) (was: 1.6.0) 1.4.2

[jira] [Resolved] (SPARK-5161) Parallelize Python test execution

[ https://issues.apache.org/jira/browse/SPARK-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-5161. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7031

[jira] [Resolved] (SPARK-8214) math function: hex

[ https://issues.apache.org/jira/browse/SPARK-8214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8214. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6976

[jira] [Resolved] (SPARK-8610) Separate Row and InternalRow (part 2)

[ https://issues.apache.org/jira/browse/SPARK-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8610. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7003

[jira] [Commented] (SPARK-8636) CaseKeyWhen has incorrect NULL handling

[ https://issues.apache.org/jira/browse/SPARK-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604724#comment-14604724 ] Davies Liu commented on SPARK-8636: --- [~animeshbaranawal] What happen if there is null in

[jira] [Resolved] (SPARK-8686) DataFrame should support `where` with expression represented by String

[ https://issues.apache.org/jira/browse/SPARK-8686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8686. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7063

[jira] [Updated] (SPARK-8677) Decimal divide operation throws ArithmeticException

[ https://issues.apache.org/jira/browse/SPARK-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8677: -- Assignee: Liang-Chi Hsieh Decimal divide operation throws ArithmeticException

[jira] [Created] (SPARK-8680) PropagateTypes is very slow when there are lots of columns

Davies Liu created SPARK-8680: - Summary: PropagateTypes is very slow when there are lots of columns Key: SPARK-8680 URL: https://issues.apache.org/jira/browse/SPARK-8680 Project: Spark Issue

[jira] [Updated] (SPARK-8680) PropagateTypes is very slow when there are lots of columns

[ https://issues.apache.org/jira/browse/SPARK-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8680: -- Description: The time for PropagateTypes is O(N*N), N is the number of columns, which is very slow if

[jira] [Resolved] (SPARK-8583) Refactor python/run-tests to integrate with dev/run-test's module system

[ https://issues.apache.org/jira/browse/SPARK-8583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8583. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6967

[jira] [Resolved] (SPARK-5482) Allow individual test suites in python/run-tests

[ https://issues.apache.org/jira/browse/SPARK-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-5482. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6967

[jira] [Resolved] (SPARK-8620) cleanup CodeGenContext

[ https://issues.apache.org/jira/browse/SPARK-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8620. --- Resolution: Fixed Fix Version/s: 1.5.0 cleanup CodeGenContext --

[jira] [Resolved] (SPARK-8652) PySpark tests sometimes forget to check return status of doctest.testmod(), masking failing tests

[ https://issues.apache.org/jira/browse/SPARK-8652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8652. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7032

[jira] [Commented] (SPARK-8670) Nested columns can't be referenced (but they can be selected)

[ https://issues.apache.org/jira/browse/SPARK-8670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603605#comment-14603605 ] Davies Liu commented on SPARK-8670: --- I think you should use `df.stats.age` or

[jira] [Updated] (SPARK-8620) cleanup CodeGenContext

[ https://issues.apache.org/jira/browse/SPARK-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8620: -- Assignee: Wenchen Fan cleanup CodeGenContext -- Key: SPARK-8620

[jira] [Resolved] (SPARK-8635) improve performance of CatalystTypeConverters

[ https://issues.apache.org/jira/browse/SPARK-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8635. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 7018

[jira] [Resolved] (SPARK-8237) misc function: sha2

[ https://issues.apache.org/jira/browse/SPARK-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8237. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6934

[jira] [Resolved] (SPARK-8371) improve unit test for MaxOf and MinOf

[ https://issues.apache.org/jira/browse/SPARK-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8371. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6825

[jira] [Created] (SPARK-8610) Separate Row and InternalRow (part 2)

Davies Liu created SPARK-8610: - Summary: Separate Row and InternalRow (part 2) Key: SPARK-8610 URL: https://issues.apache.org/jira/browse/SPARK-8610 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-8431) Add in operator to DataFrame Column in SparkR

[ https://issues.apache.org/jira/browse/SPARK-8431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8431. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6941

[jira] [Resolved] (SPARK-8359) Spark SQL Decimal type precision loss on multiplication

[ https://issues.apache.org/jira/browse/SPARK-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8359. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6814

[jira] [Resolved] (SPARK-8190) ExpressionEvalHelper.checkEvaluation should also run the optimizer version

[ https://issues.apache.org/jira/browse/SPARK-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8190. --- Resolution: Fixed ExpressionEvalHelper.checkEvaluation should also run the optimizer version

[jira] [Created] (SPARK-8579) Support arbitrary object in UnsafeRow

Davies Liu created SPARK-8579: - Summary: Support arbitrary object in UnsafeRow Key: SPARK-8579 URL: https://issues.apache.org/jira/browse/SPARK-8579 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-8187) date/time function: date_sub

[ https://issues.apache.org/jira/browse/SPARK-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8187: -- Shepherd: Davies Liu date/time function: date_sub Key:

[jira] [Updated] (SPARK-8186) date/time function: date_add

[ https://issues.apache.org/jira/browse/SPARK-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8186: -- Shepherd: Davies Liu date/time function: date_add Key:

[jira] [Commented] (SPARK-7810) rdd.py _load_from_socket cannot load data from jvm socket if ipv6 is used

[ https://issues.apache.org/jira/browse/SPARK-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598610#comment-14598610 ] Davies Liu commented on SPARK-7810: --- What's the stack trace look like? Does the host

[jira] [Resolved] (SPARK-8492) Support BinaryType in UnsafeRow

[ https://issues.apache.org/jira/browse/SPARK-8492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8492. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6911

[jira] [Resolved] (SPARK-8307) Improve timestamp from parquet

[ https://issues.apache.org/jira/browse/SPARK-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8307. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6759

[jira] [Commented] (SPARK-8301) Improve UTF8String substring/startsWith/endsWith/contains performance

[ https://issues.apache.org/jira/browse/SPARK-8301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594926#comment-14594926 ] Davies Liu commented on SPARK-8301: --- [~rxin] Why I can't assign this JIRA to

[jira] [Resolved] (SPARK-8301) Improve UTF8String substring/startsWith/endsWith/contains performance

[ https://issues.apache.org/jira/browse/SPARK-8301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8301. --- Resolution: Fixed Fix Version/s: 1.5.0 Improve UTF8String

[jira] [Resolved] (SPARK-8422) Introduce a module abstraction inside of dev/run-tests

[ https://issues.apache.org/jira/browse/SPARK-8422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8422. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6866

[jira] [Commented] (SPARK-8477) Add in operator to DataFrame Column in Python

[ https://issues.apache.org/jira/browse/SPARK-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593928#comment-14593928 ] Davies Liu commented on SPARK-8477: --- [~rxin] [~yuu.ishik...@gmail.com] We already have

[jira] [Created] (SPARK-8492) Support BinaryType in UnsafeRow

Davies Liu created SPARK-8492: - Summary: Support BinaryType in UnsafeRow Key: SPARK-8492 URL: https://issues.apache.org/jira/browse/SPARK-8492 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-8477) Add in operator to DataFrame Column in Python

[ https://issues.apache.org/jira/browse/SPARK-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-8477: -- Fix Version/s: 1.3.0 Add in operator to DataFrame Column in Python

[jira] [Resolved] (SPARK-8477) Add in operator to DataFrame Column in Python

[ https://issues.apache.org/jira/browse/SPARK-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8477. --- Resolution: Implemented Target Version/s: 1.3.0 (was: 1.5.0) Add in operator to DataFrame

[jira] [Resolved] (SPARK-8339) Itertools islice requires an integer for the stop argument.

[ https://issues.apache.org/jira/browse/SPARK-8339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8339. --- Resolution: Fixed Fix Version/s: 1.4.1 1.5.0 Issue resolved by pull request

[jira] [Resolved] (SPARK-8444) Add Python example in streaming for queueStream usage

[ https://issues.apache.org/jira/browse/SPARK-8444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8444. --- Resolution: Fixed Fix Version/s: 1.5.0 Issue resolved by pull request 6884

[jira] [Assigned] (SPARK-8461) ClassNotFoundException when code generation is enabled

[ https://issues.apache.org/jira/browse/SPARK-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-8461: - Assignee: Davies Liu ClassNotFoundException when code generation is enabled

[jira] [Resolved] (SPARK-8207) math function: bin

[ https://issues.apache.org/jira/browse/SPARK-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-8207. --- Resolution: Fixed math function: bin -- Key: SPARK-8207

[jira] [Commented] (SPARK-8477) Add in operator to DataFrame Column in Python

[ https://issues.apache.org/jira/browse/SPARK-8477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593850#comment-14593850 ] Davies Liu commented on SPARK-8477: --- I think we can use the upper case `In`, or another

<    12   13   14   15   16   17   18   19   20   21   >