Re: Root logger configuration

On Fri, Dec 13, 2019 at 1:08 PM Robert Bradshaw wrote: > The default behavior of Python logging is > > (1) Logging handlers may get added (usually in main or very close to it). > (2) Logging goes to those handlers, or a default handler if none were > added. > > With this proposal, we would have

Re: Beam's job crashes on cluster

This is not that difficult to implement, but would be better to be done when you guys integrated native job submition for Python.   However, I need t fix this last issue, which is the crash. Any idea where I should look at?     Sent: Friday, December 13, 2019 at 2:52 PM From: "Kyle Weaver"

Re: Root logger configuration

The default behavior of Python logging is (1) Logging handlers may get added (usually in main or very close to it). (2) Logging goes to those handlers, or a default handler if none were added. With this proposal, we would have (0) A default handler gets added. (1) Logging handlers may get added

Re: Beam's job crashes on cluster

> I applied some modifications to the code to run Beam tasks on k8s cluster using spark-submit. Interesting, how does that work? On Fri, Dec 13, 2019 at 12:49 PM Matthew K. wrote: > > I'm not sure if that could be a problem. I'm *not* running snadalone > Spark. I applied some modifications to

Re: Beam's job crashes on cluster

  I'm not sure if that could be a problem. I'm *not* running snadalone Spark. I applied some modifications to the code to run Beam tasks on k8s cluster using spark-submit. Therefore, worker nodes are spawned when spark-submit is called and connect to the master, and are supposed to be destroyed

Re: Root logger configuration

I looked at the documentation for basicConfig: https://docs.python.org/3/library/logging.html#logging.basicConfig Specifically, the following line: > This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True. That would mean

Re: Beam's job crashes on cluster

Correction: should be formatted `spark://host:port`. Should follow the rules here: https://spark.apache.org/docs/latest/submitting-applications.html#master-urls On Fri, Dec 13, 2019 at 12:36 PM Kyle Weaver wrote: > You probably will want to add argument `-PsparkMasterUrl=localhost:8080` > (or

Re: Beam's job crashes on cluster

You probably will want to add argument `-PsparkMasterUrl=localhost:8080` (or whatever host:port your Spark master is on) to the job-server:runShadow command. Without specifying the master URL, the default is to start an embedded Spark master within the same JVM as the job server, rather than

Re: Beam's job crashes on cluster

Job server is running on master node by this:   ./gradlew :runners:spark:job-server:runShadow --gradle-user-home `pwd`   Spark workers (executors) run on separate nodes, sharing /tmp (1GB size) in order to be able to access Beam job's MANIFEST. I'm running Python 2.7.   There is no other

Re: Beam's job crashes on cluster

> Do workers need to talk to job server independent from spark executors? No, they don't. >From the time stamps in your logs, it looks like the sigbus happened after the executor was lost. Some additional info that might help us establish a chain of causation: - the arguments you used to start

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (binding) On Fri, Dec 13, 2019 at 10:23 AM Pablo Estrada wrote: > > +1 (binding) > > On Fri, Dec 13, 2019 at 8:47 AM Maximilian Michels wrote: >> >> +1 (binding) >> >> On 13.12.19 17:10, Jeff Klukas wrote: >> > +1 (non-binding) >> > >> > On Thu, Dec 12, 2019 at 11:58 PM Kenneth Knowles > >

Re: Root logger configuration

Thanks for looking into this. I'm not sure unconditionally calling logging.basicConfig() on module import is the correct solution--this prevents modules that wish to set up handlers in place of the default handler from being able to do so. (This is why logging.basicConfig is lazily done at the

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (binding) On Fri, Dec 13, 2019 at 8:47 AM Maximilian Michels wrote: > +1 (binding) > > On 13.12.19 17:10, Jeff Klukas wrote: > > +1 (non-binding) > > > > On Thu, Dec 12, 2019 at 11:58 PM Kenneth Knowles > > wrote: > > > > Please vote on the proposal for Beam's

Re: Python: pytest migration update

Update: 37m precommit time with the latest PR (in review). On Tue, Dec 10, 2019 at 11:21 AM Udi Meiri wrote: > > > On Mon, Dec 9, 2019 at 9:33 PM Kenneth Knowles wrote: > >> >> >> On Mon, Dec 9,

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (binding) On 13.12.19 17:10, Jeff Klukas wrote: +1 (non-binding) On Thu, Dec 12, 2019 at 11:58 PM Kenneth Knowles > wrote: Please vote on the proposal for Beam's mascot to be the Firefly. This encompasses the Lampyridae family of insects, without

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (non-binding) On Thu, Dec 12, 2019 at 11:58 PM Kenneth Knowles wrote: > Please vote on the proposal for Beam's mascot to be the Firefly. This > encompasses the Lampyridae family of insects, without specifying a genus or > species. > > [ ] +1, Approve Firefly being the mascot > [ ] -1,

Re: Beam's job crashes on cluster

Actually the reason for that error is Job Server/JRE crashes at final stages and service becomes unavailable (note: job is on a very small dataset that is the absence of cluster, will be done in a couple of seconds):   19/12/13 15:22:11 INFO ContextCleaner: Cleaned accumulator 43 19/12/13

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 (non-binding) On Fri, Dec 13, 2019 at 12:47 PM jincheng sun wrote: > +1 （non-binding） > > Alex Van Boxel 于2019年12月13日 周五16:21写道： > >> +1 >> >> On Fri, Dec 13, 2019, 05:58 Kenneth Knowles wrote: >> >>> Please vote on the proposal for Beam's mascot to be the Firefly. This >>> encompasses the

Re: Beam's job crashes on cluster

Hi Kyle,   This is the pipeleine options config (I replaced localhost with actual job server's IP address, and still receive the same error. Do workers need to talk to job server independent from spark executors?):   options = PipelineOptions([     "--runner=PortableRunner",    

Re: Committed without review. Sorry.

Thanks for being transparent about this, Pablo. :)  Best, Łukasz pt., 13 gru 2019 o 06:32 Kenneth Knowles napisał(a): > Thanks for alerting dev@. > > IMO this was handled perfectly by all parties. > > Kenn > > On Thu, Dec 12, 2019 at 4:09 PM Valentyn Tymofieiev > wrote: > >> The change LGTM,

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 （non-binding） Alex Van Boxel 于2019年12月13日 周五16:21写道： > +1 > > On Fri, Dec 13, 2019, 05:58 Kenneth Knowles wrote: > >> Please vote on the proposal for Beam's mascot to be the Firefly. This >> encompasses the Lampyridae family of insects, without specifying a genus or >> species. >> >> [ ] +1,

Testing Apache Beam with JDK 14 EA builds

Hi, I work on OpenJDK at Oracle and try to encourage popular open source projects to test their releases on latest OpenJDK Early Access builds (i.e. JDK 14 -ea, atm), by providing them with regular information [0] describing new builds, their features, and making sure that their bug reports

Re: [VOTE] Beam's Mascot will be the Firefly (Lampyridae)

+1 On Fri, Dec 13, 2019, 05:58 Kenneth Knowles wrote: > Please vote on the proposal for Beam's mascot to be the Firefly. This > encompasses the Lampyridae family of insects, without specifying a genus or > species. > > [ ] +1, Approve Firefly being the mascot > [ ] -1, Disapprove Firefly being