Hey Aaron, I think I can have Node.js compact working for you in fairly short order. If you wouldn't mind testing it I'll try to get a patch together tonight.
Best, Randy On Sat, May 3, 2014 at 6:19 PM, Aaron Mefford <aa...@mefford.org> wrote: > That does appear to be the case after being pointed at the code in Flume. > Biting off implementation of a protocol was a bit more than I felt I could > manage, especially in Node.js, a language I am very new too. I really have > no idea how complicated that would be, could be that it is not as bad as I > thought. I've been coding for a number of years in a number of other > languages, but this is the first project in Node.js that I have worked on. > > JensG on stack overflow found this for me: > http://stackoverflow.com/questions/23377767/node-js-to- > flume-ng/23383545?noredirect=1#comment35920812_23383545 > > https://issues.apache.org/jira/browse/FLUME-1894 > > | args.protocolFactory(new TCompactProtocol.Factory()); > args.inputTransportFactory(new TFastFramedTransport.Factory()); > args.outputTransportFactory(new TFastFramedTransport.Factory()); > args.processor(new ThriftSourceProtocol.Processor<ThriftSourceHandler>(new > ThriftSourceHandler()));| > > From the Java Flume source, ThriftSource.java. The FastFramedTransport is > wire compatible with Framed from what I understand, so that is not an > issue. However, Compact and Binary do not seem to be compatible. > > I was getting very hopeful once I was successfully communicating with > blocking code, I didn't think the step to async would be that bad, I > assumed the framework would do the heavy lifting behind the scenes and > allow me to just register callbacks. > > Sounds like that may not be the case. > > How difficult do you expect it would be to implement TCompact in JS? I do > like the idea of native support, and am happy to do it if it is something I > can do in a reasonable time frame. > > Aaron > > > On 5/3/14, 12:36 PM, Randy Abernethy wrote: > >> Hey Aaron, >> >> So is it true that Flume-NG is requiring compact protocol for Thrift >> clients? >> >> If that is so I think the easiest way forward might be to add the compact >> protocol to the Node.js library in Node directly. Particularly because the >> Node.js end point code is not designed like the end point layers in other >> supported languages (the async bit being the key factor). >> >> I would be happy to help you get this sorted either way. >> >> -Randy >> >> >> On Sat, May 3, 2014 at 9:33 AM, Aaron Mefford <aa...@mefford.org> wrote: >> >> I am trying to get communication setup between Node.JS and Flume-NG over >>> thrift. I seem to keep running up against brick walls in the process >>> however. The first of those was that all of the existing projects that >>> seemed to do this were seemingly abandoned about 3 years ago. The next >>> problem came when I found that the javascript stubs for Thrift do not >>> support the CompactProtocol. >>> >>> I finally settled on needing to write a C++ extension for node, and have >>> proceeded in that vein. I have now made it to the point where I can >>> send a >>> message from javascript to Flume-NG using thrift in synchronous mode. >>> However as Node.Js needs non-blocking IO I now need to take the next >>> step >>> and implement it with callbacks in async mode. I was at a complete loss >>> yesterday until I found one lone web post referring to the ability to >>> request the stubs be built with cob support, which I have now done and >>> feel >>> like I am soo close, but I cannot see how to construct the objects >>> required >>> to do so. There seems to be absolutely no documentation or code samples >>> available for this on the web, at least according to the Google. I >>> assume >>> the archives of this list are included in Google, and as such have not >>> specifically consulted them. If my question is largely answered within >>> those, I will find another way to search them. >>> >>> At this point it looks like my stub constructor needs and TAsyncChannel, >>> and a TProtocolFactory for it's contstructor. While I think I can see >>> how >>> to get the TProtocolFactory, I am not sure how to provide a Factory that >>> will contstruct Protocols with the proper TTransport, TFramedTransport. >>> With the TAsyncChannel it seems all the classes provided in the >>> distribution are virtual, and there was not one generated from the >>> .thrift >>> file in the stubs. Is it expected for the user to implement the >>> TAsyncChannel? >>> >>> Can anyone point me in the right direction on this? I feel like I am so >>> close, but just missing something, it has been a very long week trying to >>> solve this communication need. >>> >>> Aaron >>> >>> >
