Hi Adam,
thanks for your reply! I am currently using something quite
similar as you do, but with lambda functions and templates.
Its not complete yet, because I'm missing the closing of socket.
Currently its just a retry on the actual API methods.
Attached is the rough idea: I use a template<typename Func>
RetryAPICall(Func aAPIMethod) that accepts a function *with*
parameters, and executes this function in a try-catch-block.
Due to this construct, the functions can have arbitrarily many
parameters, and the can return values by reference.
The try-catch is wrapped in a incremental backoff loop with a
number of retries (set the variable mMaxRetries).
The actual lambda function is constructed from the RPC method
via the following construct (all parameters passed as reference):
auto fAPIMethod = [&](){ mThriftAPI->UserPassLogin(mSessionId, aUsername,
aPassword); };
Then the retry-aware API call can be executed simply with:
RetryAPICall(fAPIMethod);
Does the attached code make sense to you? I've just hacked it
together yesterday, so it might not be perfect yet. Let me know
if you need more context to make sense of it.
All the best,
Mario
On 03.10.2017 16:24, Adam Marcionek wrote:
> Hi Mario,
>
> We're pretty new to thrift, so take this advice with a grain of salt. Our
> C++ client's generic retry functionality on ANY thrift RPC call is as the
> following pseudocode:
>
> catch TTransportException& tx
> If the TTransportExceptionType (tx.getType()) is UNKNOWN, NOT_OPEN or
> TIMEDOUT, we close the transport, reset the socket, transport and client and
> reopen the transport. Then retry the command.
> Else (getType() is BAD_ARGS, CORRUPTED_DATA, END_OF_FILE, INTERNAL_ERROR,
> INTERRUPTED) just retry the command.
> catch (TException& tx)
> Simply fail the call (i.e. do nothing with retry or
> reconnect.)
>
> After a retry failure, we simply fail the call and let higher level code
> handle it. This is all done from a macro. If you'd like that code, send me
> an email and I'll reply with it.
>
> I would actually LOVE to see C++ lamba function do this so if you have
> something, please share!
>
> Adam
>
>
>
> From: Mario Emmenlauer [mailto:[email protected]]
> Sent: Tuesday, October 3, 2017 6:14 AM
> To: [email protected]
> Subject: best way to wrap a retry in C++ client?
>
>
> Dear All,
>
> I'm super happy with Thrift, for its speed and exception handling!
> I've developed a Java server and C++ client.
>
> However there are occasional failures to connect (and disconnects)
> beyond my understanding. Its possible that the server is just busy,
> and a retry would help. To implement a generic retry in C++ I've
> found that lambda functions are useful. But I'm curious what other
> people are using and/or if there are ready-made solutions for this
> problem? I guess most thrift users face the same issue?
>
> On a related note, I've found the excellent discussion of "Designing
> robust distributed systems" from JensG at stackoverflow:
> https://stackoverflow.com/questions/23013942/handling-failures-in-thrift-in-general
>
> Its from 2014. Is there an implementation of such patterns? What do
> people commonly use for retry and associated issues?
>
> All the best,
>
> Mario Emmenlauer
>
>
> --
> BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203
> Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
> D-81669 München http://www.biodataanalysis.de/
>
> ________________________________
>
Viele Gruesse,
Mario Emmenlauer
--
BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203
Balanstr. 43 mailto: memmenlauer * biodataanalysis.de
D-81669 München http://www.biodataanalysis.de/
//
// Developed by: Mario Emmenlauer ([email protected])
// Balanstrasse 43, 81669 Munich
// http://www.biodataanalysis.de/
//
#include <thrift/transport/TSSLSocket.h>
#include <thrift/transport/TSocket.h>
#include <thrift/transport/TTransportUtils.h>
#include <thrift/transport/TTransportException.h>
#include <thrift/protocol/TBinaryProtocol.h>
#include <iostream>
#include <fstream>
#include <memory>
#include <string>
#include <stdlib.h>
#include <exception>
#include <stdexcept>
#include <chrono>
#include <thread>
template<typename Func>
void
ThriftAPIClient::
RetryAPICall(Func aAPIMethod) {
size_t vGrowingSleepTimeMilliSeconds = 1;
for (size_t vTryIdx = 1; vTryIdx <= mMaxRetries; ++vTryIdx) {
try {
// invoke the actual API method:
aAPIMethod();
// if the API call was successful, abort the retry-loop:
break;
} catch (const apache::thrift::transport::TTransportException& vTTransportException) {
if (vTryIdx == mMaxRetries) {
// the retries are over, re-throw the exception:
std::cerr << "ThriftAPIClient::RetryAPICall(): the retries are over, will throw." << std::endl;
throw;
}
// if we catch a TTransportException, it might be a retryable error:
const apache::thrift::transport::TTransportException::TTransportExceptionType vTTransportExceptionType = vTTransportException.getType();
if (vTTransportExceptionType == apache::thrift::transport::TTransportException::TTransportExceptionType::UNKNOWN ||
vTTransportExceptionType == apache::thrift::transport::TTransportException::TTransportExceptionType::NOT_OPEN ||
vTTransportExceptionType == apache::thrift::transport::TTransportException::TTransportExceptionType::TIMED_OUT ||
vTTransportExceptionType == apache::thrift::transport::TTransportException::TTransportExceptionType::END_OF_FILE ||
vTTransportExceptionType == apache::thrift::transport::TTransportException::TTransportExceptionType::INTERRUPTED) {
// The TTransportException is retriable:
std::cerr << "ThriftAPIClient::RetryAPICall(): caught a retryable TTransportException, will sleep for " << vGrowingSleepTimeMilliSeconds << "ms and retry." << std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(vGrowingSleepTimeMilliSeconds));
// increase the sleep time for the next iteration of retries:
vGrowingSleepTimeMilliSeconds *= 2;
} else {
// The TTransportException is *not* retriable:
std::cerr << "ThriftAPIClient::RetryAPICall(): caught a non-retryable TTransportException (type " << vTTransportExceptionType << "), will throw." << std::endl;
throw;
}
}
}
}
void
ThriftAPIClient::
Login(const std::string& aUsername, const std::string& aPassword) {
auto fAPIMethod = [&](){ mThriftAPI->Login(mSessionId, aUsername, aPassword); };
RetryAPICall(fAPIMethod);
}
void
ThriftAPIClient::
AddUser(const std::string& aUsername, const std::string& aPassword) {
auto fAPIMethod = [&](){ mThriftAPI->AddUser(mSessionId, aUsername, aPassword); };
RetryAPICall(fAPIMethod);
}
