Re: not able to launch dataflow job - permission issues with service account

As the message says, enable the api from cloud console of gcloud cli utility. (Refer dataflow docs) If that’s enabled, ensure the robot account has the permissions it needs - dataflow api, gcs and anything else you are trying to access. On Thu, Nov 22, 2018 at 04:33 Unais T wrote: > I am

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

+1 Sent from my iPhone > On Sep 17, 2018, at 15:26, Ankur Goenka wrote: > > +1 > >> On Sun, Sep 16, 2018 at 3:20 AM Maximilian Michels wrote: >> +1 (binding) >> >> On 15.09.18 20:07, Reuven Lax wrote: >> > +1 >> > >> > On Sat, Sep 15, 2018 at 9:40 AM Rui Wang > >

Re: rename: BeamRecord -> Row

++ On Fri, Feb 2, 2018 at 1:33 PM Rafael Fernandez wrote: > Very strong +1 > > > On Fri, Feb 2, 2018 at 1:24 PM Reuven Lax wrote: > >> We're looking at renaming the BeamRecord class >> , that was used for columnar

Unable to use MapState with DoFnTester

va:195) at org.apache.beam.sdk.transforms.DoFnTester.(DoFnTester.java:704) at org.apache.beam.sdk.transforms.DoFnTester.of(DoFnTester.java:92) at com.brightcove.rna.transforms.functions.GenerateMutationsFnTest.testDefaultTimestamp(GenerateMutationsFnTest.java:42) Is MapState supported with DoFnTester? -- Ankur Chauhan

Re: [PROPOSAL] Beam Go SDK feature branch

Hi This is amazing. Having used Java ask for over two years now and recently transitioned to writing go for a bunch of microservices, I really like the simplicity. I used gleam to do data processing and loved it. That said I would much rather prefer the beam sdk paradigms though. Ankur On

[grpc-io] Minimum and recommended pod requirements

Hi I'm running a grpc-java application in a kubernetes cluster on gke. This application listens for request from the user, reads google cloud bigtable over grpc using the official bigtable client and trouble with the rows scanned with some simple parsing. It seems like whenever I run the

Re: [grpc-io] [grpc-java] Header propagation in service mesh (linkerd)

Published a cleaned out version of these interceptors at - https://github.com/ankurcha/linkerd-grpc-interceptors On Sunday, July 30, 2017 at 12:42:37 AM UTC-7, Ankur Chauhan wrote: > > Thanks for the code example Spencer. This is exactly what I needed. I'll > make it into a library an

Re: [grpc-io] [grpc-java] Header propagation in service mesh (linkerd)

t;> >> I have created a branch on github that shows what I mean: >> https://github.com/grpc/grpc-java/compare/master...zpencer:grpc-io-l5d-demo?expand=1 >> >> >> >> On Thu, Jul 27, 2017 at 9:01 PM, Ankur Chauhan <an...@malloc64.com >> > wrote: >&

[grpc-io] [grpc-java] Header propagation in service mesh (linkerd)

Hi, I am trying to integrate some of our grpc services with linkerd ( a service mesh ). As a part of doing that, the recommendation is that the apps should propagate any headers with "l5d-" prefix to all downstream services. I tried looking at some tests for io.grpc.Context but can't figure

Using timers to flush elements buffered using state API

quot;flush" state in case of window closure (with or without timers). Also a general description / example of using the timer api may be helpful. -- Ankur Chauhan

Re: How to decrease latency when using PubsubIO.Read?

build()) .build(); } There is a lot of trial and error involved in getting the end-to-end latency down so I would suggest enabling the profiling using the —saveProfilesToGcs option and get a sense of what is exactly happening. — Ankur Chauhan > On May 24, 2017, at 9:09 AM, Josh <jof...@gmail

Re: How to decrease latency when using PubsubIO.Read?

What runner address you using. Google cloud dataflow uses a closed source version of the pubsub reader as noted in a comment on Read class. Ankur Chauhan Sent from my iPhone > On May 24, 2017, at 04:05, Josh <jof...@gmail.com> wrote: > > Hi all, > > I'm using Pub

Re: Reprocessing historic data with streaming jobs

I have sort of a similar usecase when dealing with failed / cancelled / broken streaming pipelines. We have an operator that continuously monitors the min-watermark of the pipeline and when it detects that the watermark is not advancing for more than some threshold. We start a new pipeline and

Cloning options instances

Hi I am attempting to build a sort of a supervisor process for dataflow pipelines that starts up dataflow jobs based on some inputs combined with some common presets. I used to use the cloneAs method on the pipelineoptions class but noticed that if has been removed and wanted to ask if there

Re: Renaming SideOutput

This question maybe obvious to others but why is there a distinction between main output and additional outputs? Why not just have a simple list of outputs where the first one is the Main one. -- AC Sent from my iPhone > On Apr 12, 2017, at 18:08, Melissa Pashniak

Re: Emitting results saved using state api without input (cleanup)

11, 2017, at 10:28 AM, Ankur Chauhan <an...@malloc64.com> wrote: > > Thanks for confirming a hunch that I had. I was considering doing that but > the javadoc saying "this feature is not implemented by any runner" sort of > put me off. > > Is there a more u

Re: Renaming SideOutput

+1 this is pretty much the topmost things that I found odd when starting with the beam model. It would definitely be more intuitive to have a consistent name. Sent from my iPhone > On Apr 11, 2017, at 18:29, Aljoscha Krettek wrote: > > +1 > >> On Wed, Apr 12, 2017, at

Re: Emitting results saved using state api without input (cleanup)

Thanks for confirming a hunch that I had. I was considering doing that but the javadoc saying "this feature is not implemented by any runner" sort of put me off. Is there a more up to date list of similar in progress features? If not it may be helpful to keep one. Thanks! Ank

Emitting results saved using state api without input (cleanup)

e session is only filled with error inputs, I would keep on buffering the input and never emit anything. My question is, how do i emit this buffer event when there is no more input? — Ankur Chauhan

[grpc-io] [grpc-java] Lots of Stream Error messages while doing Server-side streaming RPC

WithFlowControl(rowStream.iterator(), responseObserver); Can someone elaborate on the origin on these error messages? They seem bad but I can't seem to find how to control them. -- Ankur Chauhan -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To un

Re: [grpc-io] Re: grpc + appengine? Is this possible?

gt;>>>> hi, >>>>> >>>>> Any updates on golang grpc support on app-engine? >>>>> Are we looking at months or a year? >>>>> >>>>> On Wednesday, March 30, 2016 at 3:46:26 AM UTC-7, Miguel Vitorino >>>>> wro

[jira] [Commented] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326822#comment-15326822 ] Ankur Chauhan commented on SPARK-6707: -- Yes that is exactly what was what is wanted to express

Re: Mesos/Marathon/HAProxy Logging

This may help: http://serverfault.com/questions/331079/haproxy-and-forwarding-client-ip-address-to-servers We use similar options to ensure we have the remote ip. On 25 Aug 2015, at 09:30, John Omernik j...@omernik.com wrote: I have been playing with an application that is a very simple

Re: MesosCon Seattle attendee introduction thread

Hi all, I am Ankur Chauhan. I am a Sr. Software engineer with the Reporting and Analytics team at Brightcove Inc. I have been evaluating, tinkering, developing with mesos for about an year now. My latest adventure has been in the spark mesos integration and writing the new apache flink - mesos

Spark streaming and session windows

. Any help would be appreciated. -- Ankur Chauhan signature.asc Description: Message signed with OpenPGP using GPGMail

[Spark Streaming] Session based windowing like in google dataflow

. Any help would be appreciated. -- Ankur Chauhan signature.asc Description: Message signed with OpenPGP using GPGMail

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Pretty awesome piece. Sent from my iPhone On Aug 5, 2015, at 10:10, Hawin Jiang hawin.ji...@gmail.com wrote: Great job, Guys Let me read it carefully. On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen se...@apache.org wrote: I forgot the link ;-)

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Pretty awesome piece. Sent from my iPhone On Aug 5, 2015, at 10:10, Hawin Jiang hawin.ji...@gmail.com wrote: Great job, Guys Let me read it carefully. On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen se...@apache.org wrote: I forgot the link ;-)

[jira] [Commented] (FLINK-1984) Integrate Flink with Apache Mesos

[ https://issues.apache.org/jira/browse/FLINK-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645190#comment-14645190 ] Ankur Chauhan commented on FLINK-1984: -- I just opened a pull request https

Questions about framework development - (HA and reconciling state)

Hi all, I am working on creating an integration between Apache Flink (http://flink.apache.org) and mesos which would be similar to the way the current hadoop-mesos integration works using the java mesos client. My current idea is that the scheduler will also run a JobManager process (similar

Re: use S3-Compatible Storage with spark

The endpoint is the property you want to set. I would look at the source for that. Sent from my iPhone On Jul 17, 2015, at 08:55, Sujit Pal sujitatgt...@gmail.com wrote: Hi Schmirr, The part after the s3n:// is your bucket name and folder name, ie

[jira] [Comment Edited] (SPARK-8009) [Mesos] Allow provisioning of executor logging configuration

[ https://issues.apache.org/jira/browse/SPARK-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616085#comment-14616085 ] Ankur Chauhan edited comment on SPARK-8009 at 7/7/15 2:42 AM

[jira] [Commented] (SPARK-8009) [Mesos] Allow provisioning of executor logging configuration

[ https://issues.apache.org/jira/browse/SPARK-8009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616085#comment-14616085 ] Ankur Chauhan commented on SPARK-8009: -- I think the correct way to handle this sort

[jira] [Commented] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576101#comment-14576101 ] Ankur Chauhan commented on SPARK-6707: -- Hi Kapil, The patch is pretty much ready

Re: Cluster autoscaling in Spark+Mesos ?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, @Sharma - Is mantis/fenzo available on github or something, I did find some maven artifacts but the repository netflix/fenzo is a 404. I am interested in learning about the bin packing logic of fenzo. - -- Ankur Chauhan On 04/06/2015 22:35

Re: Cluster autoscaling in Spark+Mesos ?

into refreshed Spark context's which are aware of the new cluster topology? On Thu, Jun 4, 2015 at 4:23 PM, Ankur Chauhan an...@malloc64.com mailto:an...@malloc64.com wrote: AFAIK Mesos does not support host level auto-scaling because that is not the scope of the mesos-master or mesos-slave. In EC2

Re: [DISCUSS] Renaming Mesos Slave

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 master/slave James made some very good points and there is no technical reason for wasting time on this. On 04/06/2015 08:45, James Vanns wrote: +1 master/slave, no change needed. I couldn't agree more. This is a barmy request; master/slave

RE: Using Spark like a search engine

Hi, I am sure you can use spark for this but it seems like a problem that should be delegated to a text based indexing technology like elastic search or something based on lucene to serve the requests. Spark can be used to prepare the data that can be fed to the indexing service. Using spark

[jira] [Commented] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547379#comment-14547379 ] Ankur Chauhan commented on SPARK-6707: -- I had not thought of that. Plus

Re: Spark on Mesos vs Yarn

to the JIRA and pull requests so that I can keep track on the progress/features. Again, thanks for replying. - -- Ankur Chauhan On 15/05/2015 00:39, Tim Chen wrote: Hi Ankur, This is a great question as I've heard similar concerns about Spark on Mesos. At the time when I started to contribute

Spark on Mesos vs Yarn

equally important, but has this changed as spark has now reached almost 1.4.0 stage? - -- Ankur Chauhan -BEGIN PGP SIGNATURE- iQEcBAEBAgAGBQJVVZKGAAoJEOSJAMhvLp3L0vEIAI4edLB2rMGk+OTI4WujxX6k Ud5NyFUpaQ8WDjOhwcWB9RK5EoM7X3wGzRcGza1HLVnvdSUBG8Ltabt47GsP2lo0 7H9y2GluUZg/RJXbN0Ehp6moWjAU1W

Re: kafka + Spark Streaming with checkPointing fails to restart

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thanks everyone, that was the problem. the create new streaming context function was supposed to setup the stream processing as well as the checkpoint directory. I had missed the whole process of checkpoint setup. With that done, everything works as

kafka + Spark Streaming with checkPointing fails to restart

6b932540ac87577e4ce8385d26699c1a7d05e/spark-console.log Could someone tell me what it causes this problem? I tried looking at the stacktrace but I am not very familiar with the codebase to make solid assertions. Any ideas as to what may be happening here. - --- Ankur Chauhan -BEGIN PGP

Re: how to monitor multi directories in spark streaming task

/2015/05/13/data.txt like this. and one new directory one day. how to create the new DStream for tomorrow’s new directory(/user/root/2015/05/13/) ?? 在 2015年5月13日，下午4:59，Ankur Chauhan achau...@brightcove.com 写道： I would suggest creating one DStream per directory and then using

Re: how to monitor multi directories in spark streaming task

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I would suggest creating one DStream per directory and then using StreamingContext#union(...) to get a union DStream. - -- Ankur On 13/05/2015 00:53, hotdog wrote: I want to use use fileStream in spark streaming to monitor multi hdfs directories,

kafka + Spark Streaming with checkPointing fails to restart

6b932540ac87577e4ce8385d26699c1a7d05e/spark-console.log Could someone tell me what it causes this problem? I tried looking at the stacktrace but I am not very familiar with the codebase to make solid assertions. Any ideas as to what may be happening here. - --- Ankur Chauhan -BEGIN PGP

data schema and serialization format suggestions

with this problem. At the high level, the requirements are fairly simple: 1. Simple and easy to understand and extend. 2. Usable in places other than spark. ( I would want to use them in other applications and tools ). 3. Ability to play nice with parquet and Kafka (nice to have). - -- Ankur Chauhan

Spark streaming updating a large window more frequently

really appreciate pointers to code samples or some blogs that could help me identify best practices. -- Ankur Chauhan signature.asc Description: Message signed with OpenPGP using GPGMail

Re: history server

Hi, Sorry this may be a little off topic but I tried searching for docs on history server but couldn't really find much. Can someone point me to a doc or give me a point of reference for the use and intent of a history server? -- Ankur On 7 May 2015, at 12:06, Koert Kuipers

Re: Nightly builds/releases?

/SPARK-1517 On May 4, 2015, at 10:25 PM, Ankur Chauhan achau...@brightcove.com wrote: Hi, Does anyone know if spark has any nightly builds or equivalent that provides binaries that have passed a CI build so that one could try out the bleeding edge without having to compile. -- Ankur

Spark + Mesos + HDFS resource split

(or EC2 instance or VM). My question is: What is the recommended resource splitting? How much memory and CPU should I preallocate for HDFS and how much should I set aside as allocatable by mesos? In addition, is there some rule-of-thumb recommendation around this? - -- Ankur Chauhan -BEGIN PGP

[jira] [Commented] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500682#comment-14500682 ] Ankur Chauhan commented on SPARK-6707: -- I have a patch (pull request: https

[jira] [Created] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

Ankur Chauhan created SPARK-6707: Summary: Mesos Scheduler should allow the user to specify constraints based on slave attributes Key: SPARK-6707 URL: https://issues.apache.org/jira/browse/SPARK-6707

[jira] [Updated] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Chauhan updated SPARK-6707: - Description: Currently, the mesos scheduler only looks at the `cpu` and `mem` resources when

[jira] [Updated] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Chauhan updated SPARK-6707: - Description: Currently, the mesos scheduler only looks at the 'cpu' and 'mem' resources when

Re: spark mesos deployment : starting workers based on attributes

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Created issue: https://issues.apache.org/jira/browse/SPARK-6707 I would really appreciate ideas/views/opinions on this feature. - -- Ankur Chauhan On 03/04/2015 13:23, Tim Chen wrote: Hi Ankur, There isn't a way to do that yet, but it's

[jira] [Updated] (SPARK-6707) Mesos Scheduler should allow the user to specify constraints based on slave attributes

[ https://issues.apache.org/jira/browse/SPARK-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Chauhan updated SPARK-6707: - Description: Currently, the mesos scheduler only looks at the `cpu` and `mem` resources when

Re: spark mesos deployment : starting workers based on attributes

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Thanks! I'll add the JIRA. I'll also try to work on a patch this weekend . - -- Ankur Chauhan On 03/04/2015 13:23, Tim Chen wrote: Hi Ankur, There isn't a way to do that yet, but it's simple to add. Can you create a JIRA in Spark

spark mesos deployment : starting workers based on attributes

such a behavior. Thanks! - -- Ankur Chauhan -BEGIN PGP SIGNATURE- iQEcBAEBAgAGBQJVHvMlAAoJEOSJAMhvLp3LaV0H/jtX+KQDyorUESLIKIxFV9KM QjyPtVquwuZYcwLqCfQbo62RgE/LeTjjxzifTzMM5D6cf4ULBH1TcS3Is2EdOhSm UTMfJyvK06VFvYMLiGjqN4sBG3DFdamQif18qUJoKXX/Z9cUQO9SaSjIezSq2gd8

Mesos - spark task constraints

(if available) and then prefer other nodes. I realize that the prefer part may not be possible but I atleast want to start with just getting them to run only on the tachyon enabled nodes. Also, if someone could give me a pointer to the mesos scheduler code in spark that'll be great. - -- Ankur Chauhan

deployment of spark on mesos and data locality in tachyon/hdfs

hdfs/tachyon cluster. - -- Ankur Chauhan -BEGIN PGP SIGNATURE- iQEcBAEBAgAGBQJVGy4bAAoJEOSJAMhvLp3L5bkH/0MECyZkh3ptWzmsNnSNfGWp Oh93TUfD+foXO2ya9D+hxuyAxbjfXs/68aCWZsUT6qdlBQU9T1vX+CmPOnpY1KPN NJP3af+VK0osaFPo6k28OTql1iTnvb9Nq+WDlohxBC/hZtoYl4cVxu8JmRlou/nb

Re: deployment of spark on mesos and data locality in tachyon/hdfs

configuration to talk to s3 or the hdfs datanode) and the mesos slave process. Is this correct? On 31/03/2015 16:43, Haoyuan Li wrote: Tachyon should be co-located with Spark in this case. Best, Haoyuan On Tue, Mar 31, 2015 at 4:30 PM, Ankur Chauhan achau...@brightcove.com mailto:achau

Re: Mesos 0.22.0

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 On 20/01/2015 20:00, Manivannan wrote: +1 On Wed, Jan 21, 2015 at 7:04 AM, Vinod Kone vinodk...@gmail.com mailto:vinodk...@gmail.com wrote: +1 @vinodkone On Jan 20, 2015, at 12:03 PM, Chris Aniszczyk z...@twitter.com

Re: [VOTE] Release Apache Mesos 0.21.1 (rc2)

be too far away from this release. If anyone else thinks otherwise please let me know. Tim On Fri, Dec 19, 2014 at 12:44 PM, Ankur Chauhan an...@malloc64.com wrote: Sorry for a late join in can we get https://issues.apache.org/jira/plugins/servlet/mobile#issue/MESOS-1711 in too

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

On Nov. 8, 2014, 8:14 a.m., Mesos ReviewBot wrote: Patch looks great! Reviews applied: [27670] All tests passed. Ankur Chauhan wrote: Bump: Is there any more review suggestion that need to be incorporated to get this patch in? Vinod Kone wrote: i was waiting

Re: Help needed

You could use HDFS of Mount NFs on all nodes and then mount that inside all the containers as a volume. Sent from my iPhone On Nov 22, 2014, at 9:40 AM, Qiang qjavaswing2...@gmail.com wrote: I having been working with docker and mesos recently and one of the app I am going to dockerize

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27670/#review60481 --- On Nov. 8, 2014, 7:28 a.m., Ankur Chauhan wrote

Re: Relative dates are parsed incorrectly.

Please have a look at http://mesos.apache.org/documentation/latest/mesos-developers-guide/ That should help you get setup/started. Github pull requests are not the suggested medium for contributing changes, you will need to submit patches to review board. Ankur Sent from my iPhone On Nov

Re: [ANN] Mesos resources searchable

WOW! That is amazing. All these things in a single place is awesome. -- Ankur On 10 Nov 2014, at 15:41, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi everyone, Quick announcement that we've added Apache Mesos and made all its resources searchable:

Re: [ANN] Mesos resources searchable

WOW! That is amazing. All these things in a single place is awesome. -- Ankur On 10 Nov 2014, at 15:41, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi everyone, Quick announcement that we've added Apache Mesos and made all its resources searchable:

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

/fetcher.cpp 400fadf94d35721cabaa9983b12a5d35f71f5b5b Diff: https://reviews.apache.org/r/27670/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

value Diffs (updated) - src/launcher/fetcher.cpp 400fadf94d35721cabaa9983b12a5d35f71f5b5b Diff: https://reviews.apache.org/r/27670/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

of a success non zero value Diffs - src/launcher/fetcher.cpp bd95928bc3191970330e839bcf41e343d5142c54 Diff: https://reviews.apache.org/r/27670/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: [VOTE] Release Apache Mesos 0.21.0 (rc1)

Hi Tim, FYI: I posted a fix for the bug pointed out by @benh at https://reviews.apache.org/r/27670/ https://reviews.apache.org/r/27670/ -- Ankur On 6 Nov 2014, at 15:31, Timothy Chen tnac...@gmail.com wrote: The Fetcher patch currently contains a bug that is going to be patched up by a new

Logging warning in mesos fetcher

I noticed in a run of bin/mesos-tests.sh --gtest_filter=FetcherTest.*: $ ./bin/mesos-tests.sh --gtest_filter=FetcherTest.* Source directory: /Users/achauhan/Projects/mesos Build directory: /Users/achauhan/Projects/mesos/build - We cannot

Re: Logging warning in mesos fetcher

Will do! -- Ankur On 5 Nov 2014, at 10:56, Vinod Kone vinodk...@gmail.com wrote: Good catch. You want to add the log initialization to the main() in fetcher.cpp. See master/main.cpp or slave/main.cpp for examples. On Wed, Nov 5, 2014 at 10:32 AM, Ankur Chauhan an...@malloc64.com wrote

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

d7754009a59fedb43e3422c56b3a786ce80164aa Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

/#comment101354 This change makes fetcher in line with slave/main.cpp or master/main.cpp and initialized the logging. - Ankur Chauhan On Nov. 5, 2014, 7:57 p.m., Ankur Chauhan wrote: --- This is an automatically generated e-mail

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

/fetcher_tests.cpp d7754009a59fedb43e3422c56b3a786ce80164aa Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

(in core-site.xml). Diffs (updated) - src/launcher/fetcher.cpp bd95928bc3191970330e839bcf41e343d5142c54 Diff: https://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

Thanks, Ankur Chauhan

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

://reviews.apache.org/r/27670/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27670: Bug fix: Check for non-zero status code and hadoop client not found

://reviews.apache.org/r/27670/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Why rely on url scheme for fetching?

the naming is as expected, then you can push the patch to reviewboard to get it reviewed and committed. On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan an...@malloc64.com mailto:an...@malloc64.com wrote: Hi, I did some learning today! This is pretty much a very rough draft of the tests

Re: Why rely on url scheme for fetching?

enough to not cause me major problems no matter when it lands. Bernd On Nov 4, 2014, at 11:44 AM, Ankur Chauhan an...@malloc64.com wrote: Hi ben, Thanks for the follow up. I think thats a perfectly fine game plan. I think I can break down things into smaller, more isolated chunks

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

On Nov. 3, 2014, 9:59 p.m., Timothy St. Clair wrote: src/launcher/fetcher.cpp, line 77 https://reviews.apache.org/r/27483/diff/1/?file=746870#file746870line77 we should probably bail here, if somehow the return is != 0 (isError() || false) Ankur Chauhan wrote: My

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

. To reply, visit: https://reviews.apache.org/r/27483/#review59842 --- On Nov. 4, 2014, 7:47 p.m., Ankur Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote: src/launcher/fetcher.cpp, line 223 https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223 Why fall through if result.isError()? Ankur Chauhan wrote: The way I am thinking of the logic of fetchers is on the lines

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

://reviews.apache.org/r/27483/diff/ Testing --- make check sudo bin/mesos-tests.sh --verbose support/mesos-style.py Thanks, Ankur Chauhan

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

On Nov. 4, 2014, 9:46 p.m., Vinod Kone wrote: src/launcher/fetcher.cpp, line 223 https://reviews.apache.org/r/27483/diff/7/?file=748389#file748389line223 Why fall through if result.isError()? Ankur Chauhan wrote: The way I am thinking of the logic of fetchers is on the lines

Re: Review Request 27483: Fetcher uses hadoop to fetch URIs regardless of the url scheme.

]);` in the `main` method? - Ankur Chauhan On Nov. 5, 2014, 6:39 a.m., Ankur Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27483

Re: Why rely on url scheme for fetching?

and/or suggestions. -- Ankur @ankurcha On 2 Nov 2014, at 18:17, Ankur Chauhan an...@malloc64.com wrote: Hi, I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` is pretty coarse grained and are more on the lines of a functional test. I was going to add some tests

Re: Why rely on url scheme for fetching?

this shouldn't surprise anyone :) If you haven't already, please read the documentation on contributing to Mesos and the style guide to ensure all the naming is as expected, then you can push the patch to reviewboard to get it reviewed and committed. On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan

Re: Why rely on url scheme for fetching?

) as Target Version = 0.22.0. Ian On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan an...@malloc64.com mailto:an...@malloc64.com wrote: Hi Tim/others, Is this to be included in the 0.21.0 release? If so, I don't know how to tag it etc. I would really (shamelessly) love it to be included

  1   2   >