[GitHub] zeppelin pull request #2206: [ZEPPELIN-1856] Flaky Test: InterpreterRestApiT...
Github user AlexanderShoshin closed the pull request at: https://github.com/apache/zeppelin/pull/2206 ---
[GitHub] zeppelin issue #2206: [ZEPPELIN-1856] Flaky Test: InterpreterRestApiTest.tes...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2206 @jongyoul it looks like I need to correct my unit tests due last changes in master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2362: [ZEPPELIN-2469] HeliumBundleFactoryTest fails on Travi...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2362 We also need to enable the `HeliumBundleFactoryTest` on CI by removing it from `-Dtests.to.exclude` variable in `.travis.yml` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2300: [ZEPPELIN-2467] Restoring second travis job functional...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2300 I think we should revert this commit then. I won't be able to fix it fast. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2300: [ZEPPELIN-2467] Restoring second travis job functional...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2300 According to @Leemoonsoo suggestion I have added a comment for the travis job configuration about excluded tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2300: [ZEPPELIN-2467] Restoring second travis job fun...
GitHub user AlexanderShoshin opened a pull request: https://github.com/apache/zeppelin/pull/2300 [ZEPPELIN-2467] Restoring second travis job functionality ### What is this PR for? Second travis job didn't work correctly. It suppose to test all core unit tests plus integration test (except for spark related test) but it was doing nothing. It was because of the exclamation mark in **-Dtest** property: ``` -Dtest='!ZeppelinSparkClusterTest,!org.apache.zeppelin.spark.*' ``` which is not supported by maven-surefire-plugin of version 2.17 (which is used in Zeppelin). Exclamation mark is supported started from 2.19 but still does not work properly. I've added plugin **exclude** configuratin instead of **-Dtest** property. After travis job was restored I had found that not all core tests worked properly. I have excluded them from the travis job for now and created jira issues: https://issues.apache.org/jira/browse/ZEPPELIN-2469 https://issues.apache.org/jira/browse/ZEPPELIN-2470 https://issues.apache.org/jira/browse/ZEPPELIN-2471 https://issues.apache.org/jira/browse/ZEPPELIN-2473 ### What type of PR is it? [Bug Fix] ### What is the Jira issue? [ZEPPELIN-2467](https://issues.apache.org/jira/browse/ZEPPELIN-2467) ### Questions: * Does the licenses files need update? **no** * Is there breaking changes for older versions? **no** * Does this needs documentation? **no** You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-2467 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2300.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2300 commit 0448c4ae7950374034c864c7ba6d4855f5ab25e8 Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-04-28T13:18:15Z added ability to exclude some unit tests using command line commit dfa332f0387232ee8bd06a9e9cb1fe3fec87c06f Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-04-28T13:22:08Z changed -Dtest flag to -Dtests.to.exclude and excluded unstable tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2206: [ZEPPELIN-1856] Flaky Test: InterpreterRestApiTest.tes...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2206 Hi @jongyoul, I've added a test for my changes but it won't run until we restore second Travis job which should start all core tests. I will write when the test will be able to run on Travis. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2233: [MINOR] Add paragraph to note
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2233 This PR is ready for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2246: [ZEPPELIN-1828] Flaky Test: RemoteInterpreterTe...
GitHub user AlexanderShoshin opened a pull request:
https://github.com/apache/zeppelin/pull/2246
[ZEPPELIN-1828] Flaky Test: RemoteInterpreterTest hanged in
zeppelin-interpreter
### What is this PR for?
When `interpreter.close()` occurs in code it shuts down
_RemoteInterpreterEventPoller_ and then tries to close thrift connection. While
closing thrift client will wait until events queue is empty. If
_RemoteInterpreterEventPoller_ has not processed all thrift events before the
shutdown signal comes, thrift client will wait forever. As a result Travis will
terminate the build after code is not responding for 10 minutes.
### What type of PR is it?
[Bug Fix]
### Todos
We need to:
* [ ] - write a test that checks that event queue is empty after
_RemoteInterpreterEventPoller_ is done,
* [ ] - clear unread events on _RemoteInterpreterEventPoller_ shutdown
signal.
### What is the Jira issue?
[ZEPPELIN-1828](https://issues.apache.org/jira/browse/ZEPPELIN-1828)
### How should this be tested?
This issue can be reprodused if you put `Thread.sleep(3000L);` to
_RemoteInterpreterEventPoller.java_
```
...
} else if (event.getType() == RemoteInterpreterEventType.OUTPUT_UPDATE_ALL)
{
Thread.sleep(3000L);
...
}
```
and run _RemoteInterpreterTest.java_.
### Questions:
* Does the licenses files need update? **no**
* Is there breaking changes for older versions? **no**
* Does this needs documentation? **no**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-1828
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2246.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2246
commit 66fe9c7c480f4dfdb5e7e00eff7198b99e2d3bac
Author: Alexander Shoshin <alexander_shos...@epam.com>
Date: 2017-04-12T08:03:54Z
test that EventPoller clear unread events on shutdown
commit 5818a83d638fdf77f0b6f146f669ad523bb0e10c
Author: Alexander Shoshin <alexander_shos...@epam.com>
Date: 2017-04-12T08:04:31Z
clear unread events on shutdown
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2233: [MINOR] Add paragraph to note
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2233 Hi Felix, thank you for looking through my PR. Method name must correspond an action it performs. Methods with names like `addSomething()` usually mean that we must pass this _something_ into the method as an input parameter. I have renamed an `addParagraph()` method because it doesn't receive a `Paragraph` instance but creates a new one instead. This also gave me an opportunity to add a new `addParagraph()` method which does receive a `Paragraph` instance. As for the `Note.addNewParagraph()` method name it seem to me that it sounds like "create new paragraph and add it to the `Note`". But if such name is still not clear maybe we should choose another one like `createNewParagraph()` or `createAndAddParagraph()` or something like this. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2233: [MINOR] Add paragraph to note
GitHub user AlexanderShoshin opened a pull request: https://github.com/apache/zeppelin/pull/2233 [MINOR] Add paragraph to note ### What is this PR for? We need to add an ability of adding custom _Paragraph_ to a _Note_. This will make it easier to write _Note_ and _Paragraph_ tests. At the moment all paragraphs are created inside _Note_ class. This refactoring will allow to write a test for ZEPPELIN-1856 issue. ### What type of PR is it? [Refactoring] ### Todos * [ ] - Rename `addParagraph()` -> `addNewParagraph()` and `insertParagraph()` -> `insertNewParagraph()`. These names will describe what mathods do: create new paragraph and add it to the note. * [ ] - Remove duplicated code from `addNewParagraph()` and `insertNewParagraph()` methods. * [ ] - Add methods `addParagraph()` and `insertParagraph()`, which receive a _Paragraph_ instance in parameters and add it to the _Note_. ### Questions: * Does the licenses files need update? **no** * Is there breaking changes for older versions? **no** * Does this needs documentation? **no** You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlexanderShoshin/zeppelin add-paragraph-to-note Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2233.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2233 commit 4391f6babe31083b143a8ae260c5feeea5b919cf Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-04-07T08:44:32Z rename addParagraph() -> addNewParagraph(), insertParagraph() -> insertNewParagraph() commit b0b1dd4841096e9090ab08f8ef55dd15c2c44698 Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-04-07T08:54:50Z remove duplicated code commit 676db154139a48a30dfc93ce19ca4937f85bae2c Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-04-07T09:02:07Z added methods addParagraph() and insertParagrepg() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2206: [ZEPPELIN-1856] Flaky Test: InterpreterRestApiTest.tes...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2206 Sure, I'll do it. But first I need to make some refactoring of `Note.java` class in separate PR. We need to be able to add a custom `Paragraph` to `Note`. This will allow to use a `Paragraph` which we can control while performing. So we will be able to catch `Paragraph` _RUNNING_ and _FINISHED_ phases in a test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2206: [ZEPPELIN-1856] Flaky Test: InterpreterRestApiT...
GitHub user AlexanderShoshin opened a pull request:
https://github.com/apache/zeppelin/pull/2206
[ZEPPELIN-1856] Flaky Test:
InterpreterRestApiTest.testRestartInterpreterPerNote
### What is this PR for?
A _NullPointerException_ occurs occasionally in
`InterpreterRestApiTest.testRestartInterpreterPerNote()` test when
`p.getResult()` returns _null_ in the following line of test method:
```
assertEquals(p.getResult().message().get(0).getData(),
getSimulatedMarkdownResult("markdown"));
```
It happens when we recieve an _OUTPUT_UPDATE_ALL_ event in
_RemoteInterpreterEventPoller_ after `RemoteInterpreter.interpret(..)` is done.
The `Paragraph.result` which already contains its final value is setted to
_null_ by `NotebookServer.onOutputClear(..)` method. As a result we recieve a
_NPE_ in assertion.
### What type of PR is it?
Bug Fix
### Todos
* [ ] - To fix this issue we need to ignore an _OUTPUT_UPDATE_ALL_ event
after job is finished.
### What is the Jira issue?
[ZEPPELIN-1856](https://issues.apache.org/jira/browse/ZEPPELIN-1856)
### Questions:
* Does the licenses files need update? **no**
* Is there breaking changes for older versions? **no**
* Does this needs documentation? **no**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-1856
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2206
commit bcf01bd999fc853e08058a77443153338f85c5d4
Author: Alexander Shoshin <alexander_shos...@epam.com>
Date: 2017-03-24T13:27:35Z
do not clear a paragraph output if the paragraph is already done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2188: [MINOR] Job parameters synchronization
GitHub user AlexanderShoshin opened a pull request: https://github.com/apache/zeppelin/pull/2188 [MINOR] Job parameters synchronization ### What is this PR for? We need to synchronize setting up job result, exception, errorMessage and dateFinished in the end of `Job.run()` method. This will guarantee that we won't see these parameters in inconsistent state in multithreading environment, e.g., if we see that a dateFinished parameter has not been set up yet we will be sure that we also don't have a job final result. This code refactoring will make it possible to resolve issue [ZEPPELIN-1856](https://issues.apache.org/jira/browse/ZEPPELIN-1856). ### What type of PR is it? Refactoring ### Todos * [ ] - Synchronize `setResult()`, `exception`, `errorMessage` and `dateFinished` in `Job.run()` method. * [ ] - Synchronize getters and setters for `exception`, `errorMessage` and `dateFinished`. ### Questions: * Does the licenses files need update? **no** * Is there breaking changes for older versions? **no** * Does this needs documentation? **no** You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlexanderShoshin/zeppelin job-results-synchronization Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2188.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2188 commit bc9bbc91918984f1141b9f946f701ad616c5fe1f Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-03-23T13:12:24Z setting job results, error, dateFinished were synchronized --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1765: [WIP] ZEPPELIN-1761. CI: ZeppelinSparkClusterTest .pyS...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1765 I had similar issue a week ago: log - https://api.travis-ci.org/jobs/201058125/log.txt?deansi=true config - https://travis-ci.org/apache/zeppelin/jobs/201058125/config --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2014: [ZEPPELIN-2065] Flaky test: NotebookTest.testScheduleP...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2014 One travis job failed - looks like known bag https://issues.apache.org/jira/browse/ZEPPELIN-1761 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2014: [ZEPPELIN-2065] Flaky test: NotebookTest.testScheduleP...
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/2014 I changed AtomicInteger to CountDownLatch to simplify the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2014: [ZEPPELIN-2065] Flaky test: NotebookTest.testSc...
GitHub user AlexanderShoshin reopened a pull request: https://github.com/apache/zeppelin/pull/2014 [ZEPPELIN-2065] Flaky test: NotebookTest.testSchedulePoolUsage ### What is this PR for? `testSchedulePoolUsage()` test checks that jobs can be executed several times in a row using a cron expression. It failed sometimes when it checked job completion by monitoring `job.dateFinished` and then tried to assert that `job.status` is `FINISHED`. It was not always true because `job.status` is changed after `job.dateFinished`. ### What type of PR is it? Bug Fix ### Todos * [ ] We need to catch job completion by checking `job.status` instead of `job.dateFinished`. * [ ] It is better to catch job completion by adding job status change listener instead of using `Thread.sleep(..)` in a test method. * [ ] We can make this test simpler by removing a check that a job will not run after a cron expressin is removed. We already have such check in a previous unit test named `testSchedule()`. ### What is the Jira issue? [ZEPPELIN-2065](https://issues.apache.org/jira/browse/ZEPPELIN-2065) ### Questions: * Does the licenses files need update? **no** * Is there breaking changes for older versions? **no** * Does this needs documentation? **no** You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-2065 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2014.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2014 commit 8a0fe02846251ec4492847daf342c29823a3304f Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-02-10T15:09:03Z use job status listener to count jobs launched by cron expression --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2014: [ZEPPELIN-2065] Flaky test: NotebookTest.testSc...
Github user AlexanderShoshin closed the pull request at: https://github.com/apache/zeppelin/pull/2014 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2014: [ZEPPELIN-2065] Flaky test: NotebookTest.testSc...
GitHub user AlexanderShoshin opened a pull request: https://github.com/apache/zeppelin/pull/2014 [ZEPPELIN-2065] Flaky test: NotebookTest.testSchedulePoolUsage ### What is this PR for? `testSchedulePoolUsage()` test checks that jobs can be executed several times in a row using a cron expression. It failed sometimes when it checked job completion by monitoring `job.dateFinished` and then tried to assert that `job.status` is `FINISHED`. It was not always true because `job.status` is changed after `job.dateFinished`. ### What type of PR is it? Bug Fix ### Todos * [ ] We need to catch job completion by checking `job.status` instead of `job.dateFinished`. * [ ] It is better to catch job completion by adding job status change listener instead of using `Thread.sleep(..)` in a test method. * [ ] We can make this test simpler by removing a check that a job will not run after a cron expressin is removed. We already have such check in a previous unit test named `testSchedule()`. ### What is the Jira issue? [ZEPPELIN-2065](https://issues.apache.org/jira/browse/ZEPPELIN-2065) ### Questions: * Does the licenses files need update? **no** * Is there breaking changes for older versions? **no** * Does this needs documentation? **no** You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-2065 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2014.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2014 commit 8a0fe02846251ec4492847daf342c29823a3304f Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2017-02-10T15:09:03Z use job status listener to count jobs launched by cron expression --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1954: [ZEPPELIN-525] Test failing in zeppelin-interpr...
Github user AlexanderShoshin closed the pull request at: https://github.com/apache/zeppelin/pull/1954 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1954: [ZEPPELIN-525] Test failing in zeppelin-interpr...
GitHub user AlexanderShoshin reopened a pull request:
https://github.com/apache/zeppelin/pull/1954
[ZEPPELIN-525] Test failing in zeppelin-interpreter
`RemoteSchedulerTest.test` was fall when CPU switched `RemoteScheduler`
execution (which executes a job in a separate thread) back to test main thread
execution after changing job status to `Status.FINISHED` but before updating
running jobs list:
```
job.setStatus(lastStatus);
// if thread execution is switched here to the main thread test will fall
synchronized (queue) {
running.remove(job);
queue.notify();
}
```
Test checked job status, saw that job was terminated and expected to see an
empty jobs runing list.
```
while (!job.isTerminated() && cycles < MAX_WAIT_CYCLES) {
Thread.sleep(TICK_WAIT);
cycles++;
}
// this assert will fail because a job was not removed from running list yet
assertEquals(0, scheduler.getJobsRunning().size());
```
But `scheduler.running` list still contained the last job and the assertion
failed.
### What is this PR for?
The goal is to synchronize updating of a job status and a scheduler running
jobs list when a job is terminated.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
[ZEPPELIN-525](https://issues.apache.org/jira/browse/ZEPPELIN-525)
### How should this be tested?
You may place a `Thread.sleep(1000);` code to the `RemoteScheduler.java`
class after the `job.setStatus(lastStatus);` (line number 354). This will
increase probability of threads switching after this line. Test should not fail.
### Questions:
* Does the licenses files need update? **no**
* Is there breaking changes for older versions? **no**
* Does this needs documentation? **no**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-525
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1954.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1954
commit 8a0f4afbdf2286748cff4cc415401670de904700
Author: Alexander Shoshin <alexander_shos...@epam.com>
Date: 2017-01-27T08:30:11Z
synchronized a job status and a list of running jobs
synchronized changing of a job status and updating a list of running jobs
after job termination
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1954: [ZEPPELIN-525] Test failing in zeppelin-interpr...
GitHub user AlexanderShoshin opened a pull request:
https://github.com/apache/zeppelin/pull/1954
[ZEPPELIN-525] Test failing in zeppelin-interpreter
`RemoteSchedulerTest.test` was fall when CPU switched `RemoteScheduler`
execution (which executes a job in a separate thread) back to test main thread
execution after changing job status to `Status.FINISHED` but before updating
running jobs list:
```
job.setStatus(lastStatus);
// if thread execution is switched here to the main thread test will fall
synchronized (queue) {
running.remove(job);
queue.notify();
}
```
Test checked job status, saw that job was terminated and expected to see an
empty jobs runing list.
```
while (!job.isTerminated() && cycles < MAX_WAIT_CYCLES) {
Thread.sleep(TICK_WAIT);
cycles++;
}
// this assert will fail because a job was not removed from running list yet
assertEquals(0, scheduler.getJobsRunning().size());
```
But `scheduler.running` list still contained the last job and the assertion
failed.
### What is this PR for?
The goal is to synchronize updating of a job status and a scheduler running
jobs list when a job is terminated.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
[ZEPPELIN-525](https://issues.apache.org/jira/browse/ZEPPELIN-525)
### How should this be tested?
You may place a `Thread.sleep(1000);` code to the `RemoteScheduler.java`
class after the `job.setStatus(lastStatus);` (line number 354). This will
increase probability of threads switching after this line. Test should not fail.
### Questions:
* Does the licenses files need update? **no**
* Is there breaking changes for older versions? **no**
* Does this needs documentation? **no**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-525
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1954.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1954
commit 8a0f4afbdf2286748cff4cc415401670de904700
Author: Alexander Shoshin <alexander_shos...@epam.com>
Date: 2017-01-27T08:30:11Z
synchronized a job status and a list of running jobs
synchronized changing of a job status and updating a list of running jobs
after job termination
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 @AhyoungRyu, thanks for the note. I've removed an url to the notebook. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 Sorry, I was not able to work on this issue during last weeks. @AhyoungRyu, I will make the corrections soon. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 @bzz, I am not sure. We already have a *Word Count* example there to describe `Flink` usage. Where should we place the link to this example? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 I am confused a bit :) @AhyoungRyu, do I need to drop my last commit or not? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 I've converted notebook to 0.6.2 format. So it should be displayed correctly now. [ZeppelinHub vew](https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL0FsZXhhbmRlclNob3NoaW4vemVwcGVsaW4vWkVQUEVMSU4tMTc4Ny9ub3RlYm9vay8yQzM1WVU4MTQvbm90ZS5qc29u) also works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 Oh, it should not looks like this. It might be another problem of new notebook json structure. I will correct this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 @zjffdu, should we do this in current PR or make a new issue for the documentation improving? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 @tae-jun, I found that **ZeppelinHub** can't display new notebooks correctly. It is because the note.json structure was changed after [ZEPPELIN-212](https://issues.apache.org/jira/browse/ZEPPELIN-212) was merged (two weeks ago). Now notebook has a `results` attribute (instead of `result`) to store paragraph results and it seems that **ZeppelinHub** can't see it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 @AhyoungRyu, thank you for your notes! * I've changed `wget` to `curl` as you suggested. * As for the error message with "permission denied" I think it is because `%sh` paragraphs with data download instructions did not finish correct. Each paragraph downloads for about 70 MB of data and unpack it then. They might have finished by timeout. In this case you don't have the `/tmp/flights98.csv` file at all. A added a recomendation to the notebook to increase `shell.command.timeout.millisecs` setting. It helped in my case. * We can't store data sets in "home" folder because `%sh` and `%flink` may be run by different users. So they may have different home folders. The `/tmp/` folder is a common folder which normally can be accessed by each user. For the case of limited access I've added `chmod 666 /tmp/flights.csv` command for each *csv* set. Maybe there is a better solution for this issue. I will be glad to receive any suggestions :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1758: [ZEPPELIN-1787] Add an example of Flink Notebook
Github user AlexanderShoshin commented on the issue: https://github.com/apache/zeppelin/pull/1758 Yes, but ZeppelinHub does not show all the paragraphs. I will try to find out the reason. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1758: [ZEPPELIN-1787] Add an example of Flink Noteboo...
GitHub user AlexanderShoshin opened a pull request: https://github.com/apache/zeppelin/pull/1758 [ZEPPELIN-1787] Add an example of Flink Notebook ### What is this PR for? This PR will add an example of batch processing with Flink to Zeppelin tutorial notebooks. There are no any Flink notebooks in the tutorial at the moment. ### What type of PR is it? Improvement ### What is the Jira issue? [ZEPPELIN-1787](https://issues.apache.org/jira/browse/ZEPPELIN-1787) ### How should this be tested? You should open `Using Flink for batch processing` notebook from the `Zeppelin Tutorial` folder and run all paragraphs one by one ### Questions: * Does the licenses files need update? - **no** * Is there breaking changes for older versions? - **no** * Does this needs documentation? - **no** You can merge this pull request into a Git repository by running: $ git pull https://github.com/AlexanderShoshin/zeppelin ZEPPELIN-1787 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1758.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1758 commit f64b60a15e2711746745df6cc78690b2fdcbe6e2 Author: Alexander Shoshin <alexander_shos...@epam.com> Date: 2016-12-13T12:52:20Z [ZEPPELIN-1787] Add an example of Flink Notebook --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
