[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214147129
  
Merging this into master, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214134616
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214039156
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214039161
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56853/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214038799
  
**[Test build #56853 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56853/consoleFull)**
 for PR 12648 at commit 
[`005892a`](https://github.com/apache/spark/commit/005892acd63b5621306f3ffd36f596ea4f204357).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214025092
  
**[Test build #56853 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56853/consoleFull)**
 for PR 12648 at commit 
[`005892a`](https://github.com/apache/spark/commit/005892acd63b5621306f3ffd36f596ea4f204357).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214024604
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-214024597
  
cc @davies


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-213892725
  
Hmmm...
```
[info] Run completed in 1 hour, 18 minutes, 30 seconds.
[info] Total number of tests run: 1805
[info] Suites: completed 65, aborted 0
[info] Tests: succeeded 1805, failed 0, canceled 0, ignored 594, pending 0
[info] All tests passed.
[info] Passed: Total 1810, Failed 0, Errors 0, Passed 1810, Ignored 594
[error] (core/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 4879 s, completed Apr 23, 2016 6:42:17 PM
[error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt 
-Pyarn -Phadoop-2.3 -Phive -Pkinesis-asl -Phive-thriftserver 
-Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest
 test ; received return code 1
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-213864239
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-213864240
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56822/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-213864100
  
**[Test build #56822 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56822/consoleFull)**
 for PR 12648 at commit 
[`005892a`](https://github.com/apache/spark/commit/005892acd63b5621306f3ffd36f596ea4f204357).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-213856851
  
# spark-shell

```

# ./bin/spark-shell
Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.0.0-SNAPSHOT
  /_/

Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 
1.8.0_45-internal)
Type in expressions to have them evaluated.
Type :help for more information.
16/04/22 23:56:50 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
Spark context available as sc (master = local[*], app id = 
local-1461369413316).
SQL context available as sqlContext.

scala>
```

# pyspark BEFORE

```
# ./bin/pyspark
Python 2.7.9 (default, Apr  2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
16/04/22 23:57:27 INFO SparkContext: Running Spark version 2.0.0-SNAPSHOT
16/04/22 23:57:28 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
16/04/22 23:57:28 INFO SecurityManager: Changing view acls to: root
16/04/22 23:57:28 INFO SecurityManager: Changing modify acls to: root
16/04/22 23:57:28 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users with view permissions: Set(root); users with 
modify permissions: Set(root)
16/04/22 23:57:29 INFO Utils: Successfully started service 'sparkDriver' on 
port 38291.
16/04/22 23:57:30 INFO Slf4jLogger: Slf4jLogger started
16/04/22 23:57:30 INFO Remoting: Starting remoting
16/04/22 23:57:30 INFO Remoting: Remoting started; listening on addresses 
:[akka.tcp://sparkDriverActorSystem@192.168.128.1:44013]
16/04/22 23:57:30 INFO Utils: Successfully started service 
'sparkDriverActorSystem' on port 44013.
16/04/22 23:57:30 INFO SparkEnv: Registering MapOutputTracker
16/04/22 23:57:30 INFO SparkEnv: Registering BlockManagerMaster
16/04/22 23:57:30 INFO DiskBlockManager: Created local directory at 
/tmp/blockmgr-d6702ea8-19ef-4c5f-b97c-1044860d5791
16/04/22 23:57:30 INFO MemoryStore: MemoryStore started with capacity 511.1 
MB
16/04/22 23:57:30 INFO SparkEnv: Registering OutputCommitCoordinator
16/04/22 23:57:31 INFO Utils: Successfully started service 'SparkUI' on 
port 4040.
16/04/22 23:57:31 INFO SparkUI: Started SparkUI at http://192.168.128.1:4040
16/04/22 23:57:31 INFO Executor: Starting executor ID driver on host 
localhost
16/04/22 23:57:31 INFO Utils: Successfully started service 
'org.apache.spark.network.netty.NettyBlockTransferService' on port 43464.
16/04/22 23:57:31 INFO NettyBlockTransferService: Server created on 43464
16/04/22 23:57:31 INFO BlockManagerMaster: Trying to register BlockManager
16/04/22 23:57:31 INFO BlockManagerMasterEndpoint: Registering block 
manager localhost:43464 with 511.1 MB RAM, BlockManagerId(driver, localhost, 
43464)
16/04/22 23:57:31 INFO BlockManagerMaster: Registered BlockManager
Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.0-SNAPSHOT
  /_/

Using Python version 2.7.9 (default, Apr  2 2015 15:33:21)
SparkContext available as sc, SQLContext available as sqlContext.
>>>
```

# pyspark AFTER
```
# ./bin/pyspark
Python 2.7.9 (default, Apr  2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: 
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/04/23 04:17:40 WARN NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
Welcome to
    __
 / __/__  ___ _/ /__
_\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.0.0-SNAPSHOT
  /_/

Using Python version 2.7.9 (default, Apr  2 2015 15:33:21)
SparkContext available as sc, SQLContext available as sqlContext.
>>>
```

# sparkR BEFORE
```
# ./bin/sparkR
R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY 

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12648#issuecomment-213856690
  
**[Test build #56822 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56822/consoleFull)**
 for PR 12648 at commit 
[`005892a`](https://github.com/apache/spark/commit/005892acd63b5621306f3ffd36f596ea4f204357).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14881][PYTHON][SPARKR] pyspark and spar...

[felixcheung]

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/12648

[SPARK-14881][PYTHON][SPARKR] pyspark and sparkR shell default log level 
should match spark-shell/Scala

## What changes were proposed in this pull request?

Change default logging to WARN for pyspark shell and sparkR shell.


## How was this patch tested?

Manually running pyspark and sparkR shell



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark pylogging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12648


commit 005892acd63b5621306f3ffd36f596ea4f204357
Author: felixcheung 
Date:   2016-04-23T04:23:35Z

default logging to WARN for pyspark and sparkR shell




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org