Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan Any suggestions?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user debugger87 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18900#discussion_r198401803
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -93,12 +93,16 @@ object CatalogStorageFormat
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan totally agree with you, i will try to regenerate output files
for `SQLQueryTestSuite`
---
-
To unsubscribe, e
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan For Hive, `desc extended dbName.table` or `desc extended
dbName.table partition (name=value)` will print `Detailed Table Information` or
`Detailed Partition Information`. Of course
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan I found that if we follow code in CatalogTable.toLinkedHashMap,
this patch will break so many test cases in
org.apache.spark.sql.SQLQueryTestSuite (compare *.out files
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cxzl25 @maropu any suggestionsï¼
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cxzl25 I run `build/mvn -Dtest=none
-DwildcardSuites=org.apache.spark.sql.streaming.FlatMapGroupsWithStateSuite
test` in my local PC, output is as follows:
```
Run completed
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cxzl25 Any suggestions?
```
org.apache.spark.sql.streaming.FlatMapGroupsWithStateSuite.flatMapGroupsWithState
- streaming with processing time timeout
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cxzl25 this PR will break some unit tests, I will fix it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user debugger87 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18900#discussion_r193730957
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -1019,6 +1021,8 @@ private[hive] object HiveClientImpl
GitHub user debugger87 reopened a pull request:
https://github.com/apache/spark/pull/18900
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
## What changes were proposed in this pull request?
Set createTime for every hive partition created in Spark SQL
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cxzl25 OK, reopen it
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
https://github.com/apache/spark/pull/19721 Fixed the same issue, i will
close it.
---
-
To unsubscribe, e-mail: reviews
Github user debugger87 closed the pull request at:
https://github.com/apache/spark/pull/18649
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@dilipbiswal
Thanks for your reply. In my eyes, there have been some mechanism or
configuration to control the number of opening files generated by SQL
Operation. e.g
Github user debugger87 closed the pull request at:
https://github.com/apache/spark/pull/18900
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
`createTime` is set by HiveMetaStore#initializeAddedPartition
```
private void initializeAddedPartition(Table tbl, PartitionIterator part,
boolean madeDir) throws MetaException
Github user debugger87 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18900#discussion_r132802873
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -97,7 +97,9 @@ object CatalogStorageFormat
Github user debugger87 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18900#discussion_r132711854
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -986,6 +986,7 @@ private[hive] object HiveClientImpl
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan Look at this PR again? I just put `createTime` into
CatalogTablePartion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan Have a look at this PR again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
My changes is not enough to support `createTime` in CatalogTablePartition,
I will check and re-commit again.
---
If your project is set up for it, you can reply to this email and have your
Github user debugger87 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18900#discussion_r132377142
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -986,6 +986,7 @@ private[hive] object HiveClientImpl
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18900
@cloud-fan could you please help me to review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user debugger87 opened a pull request:
https://github.com/apache/spark/pull/18900
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
## What changes were proposed in this pull request?
Set createTime for every hive partition created in Spark SQL
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
cc @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@cloud-fan @srowen If we can't find maintainer of hive-thriftserver in
Spark, I may have to close this PR in few days later.
---
If your project is set up for it, you can reply to this email
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@cloud-fan This patch is very simple and clear, is there any problem to
review or merge it?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@gatorsmile Could you please help me review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@jerryshao Yes, it's just copied from SQLOperation in Hive. However, those
code lines are the key point that HiveServer2 can return operation log to
client via `TFetchResultsReq` which fetchType
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@cloud-fan Any suggestions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@cloud-fan I just fix scala style issue, pls re-test again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user debugger87 commented on the issue:
https://github.com/apache/spark/pull/18649
@cloud-fan Could you please help me to review this PR? Thanks a lot!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user debugger87 opened a pull request:
https://github.com/apache/spark/pull/18649
[SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't register operation
log before execute sql statement
## What changes were proposed in this pull request?
- Add a private method
Github user debugger87 commented on the pull request:
https://github.com/apache/spark/pull/10888#issuecomment-174582117
@srowen The best choice is to close this PR and it's not necessary to do
the same work like #8943.
---
If your project is set up for it, you can reply
Github user debugger87 closed the pull request at:
https://github.com/apache/spark/pull/10888
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user debugger87 commented on the pull request:
https://github.com/apache/spark/pull/10888#issuecomment-174436205
I found another similar PR #8943 which is closed in case of that
[YARN-1963](https://issues.apache.org/jira/browse/YARN-1963) has not been
implemented. I will close
Github user debugger87 closed the pull request at:
https://github.com/apache/spark/pull/10888
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user debugger87 commented on the pull request:
https://github.com/apache/spark/pull/10888#issuecomment-174470198
@jerryshao reopen this pr ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user debugger87 reopened a pull request:
https://github.com/apache/spark/pull/10888
[SPARK-12973][YARN] Support to set priority when submit spark application
to YARN
You can merge this pull request into a Git repository by running:
$ git pull https://github.com
GitHub user debugger87 opened a pull request:
https://github.com/apache/spark/pull/10888
[SPARK-12973][YARN] Support to set priority when submit spark application
to YARN
You can merge this pull request into a Git repository by running:
$ git pull https://github.com
Github user debugger87 closed the pull request at:
https://github.com/apache/spark/pull/5089
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user debugger87 commented on the pull request:
https://github.com/apache/spark/pull/5089#issuecomment-84761276
Thanks @marmbrus and I will close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user debugger87 opened a pull request:
https://github.com/apache/spark/pull/5089
[SPARK-5387] [SQL] parquet writer runs into OOM during writing when number
of rows is large
In some extreme cases, e.g. one row has thousands of columns, this line of
code val writter
44 matches
Mail list logo
