I haven't looked closely, but it could just be a side effect of #574. I recommend you upgrade to 2.1.0 (when it releases, very very very soon now) and see if the problem goes away first.
-Joan ----- Original Message ----- From: "Carlos Alonso" <[email protected]> To: [email protected], "Joan Touzet" <[email protected]> Sent: Wednesday, August 2, 2017 8:40:39 AM Subject: Re: Understanding a couple replication errors Hi Joan, many thanks for your reply Why would the timeout and closed connections happen? I see it quite often on the logs TBH but also, a slightly different one is a timeout against the cluster itself. What may this mean? [error] 2017-08-02T12:38:27.321697Z couchdb@couch-2 <0.7114.568> -------- Replicator, request PUT to "http://admin:*****@ 127.0.0.1:5984/my_db/35e78fe9d421677c77dea34522de2b99?new_edits=false " failed due to error {error,{conn_failed,{error,etimedout}}} Regards On Mon, Jul 31, 2017 at 5:06 PM Joan Touzet < [email protected] > wrote: Hi Carlos, This issue (the {{nocatch,{mp_parser_died,noproc}}) is recently identified and patched! https://github.com/apache/couchdb/issues/574 The other error just looks like a disconnection. You may want to wait (just a couple more days!) for 2.1.0 which has all sorts of new replication features and try again. -Joan ----- Original Message ----- From: "Carlos Alonso" < [email protected] > To: "user" < [email protected] > Sent: Monday, July 31, 2017 9:58:09 AM Subject: Re: Understanding a couple replication errors On top of those, this one happens from time to time as well: [error] 2017-07-31T13:41:50.258423Z couchdb@couch-2 emulator -------- Error in process <0.16620.2434> on node 'couchdb@couch-2' with exit value: {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},... [error] 2017-07-31T13:41:50.260569Z couchdb@couch-2 <0.32053.2433> -------- Replicator, request PUT to "http://admin:*****@ 127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false " failed due to error {error, {'EXIT', {{{nocatch,{mp_parser_died,noproc}}, [{couch_att,'-foldl/4-fun-0-',3, [{file,"src/couch_att.erl"},{line,591}]}, {couch_att,fold_streamed_data,4, [{file,"src/couch_att.erl"},{line,642}]}, {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]}, {couch_httpd_multipart,atts_to_mp,4, [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}, {gen_server,call, [<0.11924.2435>, {send_req, {{url, " http://admin:[email protected]:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false ", "127.0.0.1",5984,"admin","password", "/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false", http,ipv4_address}, [{"Accept","application/json"}, {"Content-Length",32845}, {"Content-Type", "multipart/related; boundary=\"46b37c9ba185ac67bf24ee40bcc38828\""}, {"User-Agent","CouchDB-Replicator/2.0.0"}], put, {#Fun<couch_replicator_api_wrap.11.79455979>, {<<"{\"_id\":\"cba0aa16e347ac7d6b09732032828704\",\"_rev\":\"8-b90e6bd0b6281f303e587f4b9538a0af\",...\"_revisions\":{\"start\":8,\"ids\":[\"b90e6bd0b6281f303e587f4b9538a0af\",\"df66a380ea1258243bfd8a33506a18b4\",\"4cd932d1b71cebc4a55f91331d97f902\",\"a4dabd1dbf2b1de8987119a65cb44131\",\"82cc46c88ca768ff092de95c338bc0e9\",\"b4a7562ac8a831f46a194e97ea33e527\",\"d4fd4ddb998e4022e0335f609ed1ef50\",\"118cae6fc6864b3c48f788e78705c8cb\"]},\"_attachments\":{\"FF01-0020941.pdf\":{\"content_type\":\"application/pdf\",\"revpos\":8,\"digest\":\"md5-IT8Ms2HDG9EFLvfIxYTaVw==\",\"length\":21888,\"follows\":true},\"FF01-0020941.xml\":{\"content_type\":\"application/xml\",\"revpos\":8,\"digest\":\"md5-KJIXHDQbcJ3xeLaOUh11cw==\",\"length\":8526,\"follows\":true}}}">>, [{att,<<"FF01-0020941.pdf">>,<<"application/pdf">>, 21888,21888, <<33,63,12,179,97,195,27,209,5,46,247,200,197,132, 218,87>>, 8, {follows,<0.31188.2433>,#Ref<0.0.12124.197441>}, identity}, {att,<<"FF01-0020941.xml">>,<<"application/xml">>,8526, 8526, <<40,146,23,28,52,27,112,157,241,120,182,142,82,29, 117,115>>, 8, {follows,<0.31188.2433>,#Ref<0.0.12124.197441>}, identity}], <<"46b37c9ba185ac67bf24ee40bcc38828">>,32845}}, [{response_format,binary}, {inactivity_timeout,30000}, {socket_options,[{keepalive,true},{nodelay,false}]}], infinity}}, infinity]}}}} Thanks! On Mon, Jul 31, 2017 at 11:52 AM Carlos Alonso < [email protected] > wrote: > Hi all!! > > I've been observing logs for a while and trying to understand why a few > errors happen while (pull) replication is happening and also trying to > understand the consequences. > > ** connection_closed* > > Log: > ``` > [error] <0.15968.1946> -------- Replicator, request GET to > "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"; > > failed due to error connection_closed > ``` > > Cause: This one happens from time to time. It looks like the source is > closing connection, but why? > Consequences: None. I think replicator just retries the task. > > * *changes_reader_died* > > Log: > ``` > [error] <0.15968.1946> -------- Replicator, request GET to > "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"; > > failed due to error connection_closed > > [error] <0.13952.1946> -------- ChangesReader process died with reason: > {http_request_failed,"GET"," > https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed > > <https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000%22,%7Berror,connection_closed> > > }} > > [error] <0.13952.1946> -------- Replication > `363ed31e35361d024d22e0a990ca1ae5+continuous` > (`https://admin:*****@source.ip:443/my_db/` > -> `http://admin:*****@ 127.0.0.1:5984/my_db/` > < http://127.0.0.1:5984/my_db/ >) failed: changes_reader_died > > [error] <0.30404.6252> -------- Replicator, request GET to > "https://admin:*****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"; > > failed due to error {error,closing_on_request} > > [error] <0.352.0> -------- Error in replication > `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document > `28d973af08931f79fa423b54a8004b31`): changes_reader_died > Restarting replication in 5 seconds. > > [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with > reason: changes_reader_died > last msg: > {'EXIT',<0.15968.1946>,{http_request_failed,"GET"," https://admin : > *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}} > > state: > [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb," > https://admin : > *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb," > http://admin : > *****@ 127.0.0.1:5984/my_db/ > ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>}," > https://admin : > *****@source.ip:443/my_db/","http://admin:*****@ 127.0.0.1:5984/my_db/ > ",{httpdb," https://admin : > *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb," > http://admin : > *****@ 127.0.0.1:5984/my_db/ > ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri, > > 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59 > GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri, > > 28 Jul 2017 09:46:48 > GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}] > > > [error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>) > with 2 neighbors exited with reason: changes_reader_died at > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); > initial_call: {couch_replicator,init,['Argument__1']}, ancestors: > [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [], > links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...], > dictionary: > [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...], > > trap_exit: true, status: running, heap_size: 4185, stack_size: 27, > reductions: 26502159 > > [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child > "363ed31e35361d024d22e0a990ca1ae5+continuous" started with > {gen_server,start_link,undefined} at <0.13952.1946> exit with reason > changes_reader_died in context child_terminated > > [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with > reason: changes_reader_died > last msg: {'EXIT',<0.13952.1946>,changes_reader_died} > state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb," > https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ > 127.0.0.1:5984/my_db/ > ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\ > > ",...}} > > [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with > reason: changes_reader_died > last msg: {'EXIT',<0.13952.1946>,changes_reader_died} > state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb," > https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ > 127.0.0.1:5984/my_db/ > ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\ > > ",...],...}} > > [error] <0.16617.1946> -------- CRASH REPORT Process (<0.16617.1946>) > with 15 neighbors exited with reason: changes_reader_died at > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], > messages: [], links: > [<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...], > dictionary: > [{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}], > trap_exit: true, status: running, heap_size: 4185, stack_size: 27, > reductions: 6921530028 > > [error] <0.16155.1946> -------- CRASH REPORT Process (<0.16155.1946>) > with 15 neighbors exited with reason: changes_reader_died at > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], > messages: [], links: > [<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...], > dictionary: > [{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}], > trap_exit: true, status: running, heap_size: 6772, stack_size: 27, > reductions: 6914251593 > > [error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with > reason: changes_reader_died > last msg: {'EXIT',<0.13952.1946>,changes_reader_died} > state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb," > https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ > 127.0.0.1:5984/my_db/ > ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0 > > }} > > [error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with > reason: changes_reader_died > last msg: {'EXIT',<0.13952.1946>,changes_reader_died} > state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb," > https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\ > > ",,...],...}} > > [error] <0.18469.1946> -------- CRASH REPORT Process (<0.18469.1946>) > with 15 neighbors exited with reason: changes_reader_died at > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], > messages: [], links: > [<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...], > dictionary: > [{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}], > trap_exit: true, status: running, heap_size: 6772, stack_size: 27, > reductions: 6918077761 > > [error] <0.10009.1946> -------- CRASH REPORT Process (<0.10009.1946>) > with 15 neighbors exited with reason: changes_reader_died at > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], > messages: [], links: > [<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...], > dictionary: > [{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}], > trap_exit: true, status: running, heap_size: 6772, stack_size: 27, > reductions: 6919159795 > ``` > > Cause: This one looks similar to the one before. The trigger seems to be > that the source closes the connection for some reason, but the logs are > much more verbose this time. > Consequences: None. I think replicator just retries the task. > > Furthermore, this last logs show the username and passwords plain text on > the state key. > > Sorry for the wall of text. > > Regards > > -- > [image: Cabify - Your private Driver] < http://www.cabify.com/ > > > *Carlos Alonso* > Data Engineer > Madrid, Spain > > [email protected] > > Prueba gratis con este código > #CARLOSA6319 < https://cabify.com/i/carlosa6319 > > [image: Facebook] < http://cbify.com/fb_ES >[image: Twitter] > < http://cbify.com/tw_ES >[image: Instagram] < http://cbify.com/in_ES > >[image: > Linkedin] < https://www.linkedin.com/in/mrcalonso > > -- [image: Cabify - Your private Driver] < http://www.cabify.com/ > *Carlos Alonso* Data Engineer Madrid, Spain [email protected] Prueba gratis con este código #CARLOSA6319 < https://cabify.com/i/carlosa6319 > [image: Facebook] < http://cbify.com/fb_ES >[image: Twitter] < http://cbify.com/tw_ES >[image: Instagram] < http://cbify.com/in_ES >[image: Linkedin] < https://www.linkedin.com/in/mrcalonso > -- Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su destinatario, pudiendo contener información confidencial sometida a secreto profesional. No está permitida su reproducción o distribución sin la autorización expresa de Cabify. Si usted no es el destinatario final por favor elimínelo e infórmenos por esta vía. This message and any attached file are intended exclusively for the addressee, and it may be confidential. You are not allowed to copy or disclose it without Cabify's prior written authorization. If you are not the intended recipient please delete it from your system and notify us by e-mail. -- Cabify - Your private Driver Carlos Alonso Data Engineer Madrid, Spain [email protected] Prueba gratis con este código #CARLOSA6319 FacebookTwitterInstagramLinkedin Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su destinatario, pudiendo contener información confidencial sometida a secreto profesional. No está permitida su reproducción o distribución sin la autorización expresa de Cabify. Si usted no es el destinatario final por favor elimínelo e infórmenos por esta vía. This message and any attached file are intended exclusively for the addressee, and it may be confidential. You are not allowed to copy or disclose it without Cabify's prior written authorization. If you are not the intended recipient please delete it from your system and notify us by e-mail.
