Thanks Eron & Fabian.
The issue was hitting a yarn proxy url vs the node itself. For example this
worked
http://
{ip}:37716/jobs/1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/target-directory/s3%3A%2F%2F%2Fremit-flink
But this did not
http://
{ip}:20888/proxy/application_1504649135200_0001/jobs/1a0fd176ec8aabb9b8464fa481f755f0/cancel-with-savepoint/target-directory/s3%3A%2F%2F%2Fremit-flink
It's a bit confusing because the cancel api works with either and the proxy
url sometimes works as this was successful http://
{ip}:20888/proxy/application_1504649135200_0001/jobs/cca2dd609c716a7b0a195700777e5b1f/cancel-with-savepoint/target-directory/tmp/
Cheers,
Emily
On Tue, Sep 19, 2017 at 2:37 PM, Eron Wright <eronwri...@gmail.com> wrote:
> Good news, it can be done if you carefully encode the target directory
> with percent-encoding, as per:
> https://tools.ietf.org/html/rfc3986#section-2.1
>
> For example, given the directory `s3:///savepoint-bucket/my-awesome-job`,
> which encodes to `s3%3A%2F%2F%2Fsavepoint-bucket%2Fmy-awesome-job`, I was
> able to submit the following URL:
> http://localhost:8081/jobs/5c360ded6e4b7d8db103e71d68b7c8
> 3d/cancel-with-savepoint/target-directory/s3%3A%2F%2F%
> 2Fsavepoint-bucket%2Fmy-awesome-job
>
> And see the following in the log:
> 2017-09-19 14:27:45,939 INFO org.apache.flink.runtime.jobmanager.JobManager
> - Trying to cancel job 5c360ded6e4b7d8db103e71d68b7c83d
> with savepoint to s3:///savepoint-bucket/my-awesome-job
>
> -Eron
>
> On Tue, Sep 19, 2017 at 1:54 PM, Fabian Hueske <fhue...@gmail.com> wrote:
>
>> Hi Emily,
>>
>> thanks for reaching out.
>> I'm not familiar with the details of the Rest API but Ufuk (in CC) might
>> be able to help you.
>>
>> Best, Fabian
>>
>> 2017-09-19 10:23 GMT+02:00 Emily McMahon <emil...@remitly.com>:
>>
>>> I've tried every combination I can think of to pass an s3 path as the
>>> target directory (url encode, include trailing slash, etc)
>>>
>>> I can successfully pass a local path as the target directory (ie
>>> /jobs/$jobID/cancel-with-savepoint/target-directory/tmp) so I don't
>>> think there's a problem with the jobId or rest of the url. I also verified
>>> I can create the savepoint on s3 from the command line so it's not a
>>> permission issue.
>>>
>>> Here's the same question on stack overflow
>>> <https://stackoverflow.com/questions/45844298/send-flink-job-manager-a-fully-qualified-path-to-a-savepoint-directory-using-the>
>>> (with
>>> the exception that they are getting a 502 whereas I'm getting a 404)
>>>
>>> using Flink 1.3.1
>>>
>>> Anyone have a working example?
>>>
>>> Thanks,
>>> Emily
>>>
>>
>>
>
