Avro C
Hello, Is anyone out there actually using Avro C? If so, are you using it in production? I'm wondering what impact some changes that are happening for 1.4 might have on any actual users... Thanks, - Bruce
Re: new Avro committer: Bruce Mitchener
Thank you! I look forward to continuing to contribute. I filed an ICLA last week and got an email that it had been received and processed. That said, I do not yet see my name on http://people.apache.org/~jim/committers.html - Bruce On Tue, Apr 6, 2010 at 2:50 PM, Doug Cutting wrote: > The Hadoop PMC has voted to add Bruce Mitchener as an Avro committer. > > Bruce, can you please file an Individual Contributor License Agreement > (ICLA) with Apache? Instructions are at: > > http://www.apache.org/licenses/#clas > > Once this is on file I can ask for your account to be created. > > Congratulations, Bruce! > > Doug >
Thoughts on an RPC protocol
I'm assuming that the goals of an optimized transport for Avro RPC are something like the following: * Framing should be efficient, easy to implement. * Streaming of large values, both as part of a request and as a response is very important. * Being able to have multiple concurrent requests in flight, while also being able to have ordering guarantees where desired is necessary. * It should be easy to implement this in Java, C, Python, Ruby, etc. * Security is or will be important. This security can include authorization as well as privacy concerns. I'd like to see something based largely upon RFC 3080, with some simplifications and extensions: http://www.faqs.org/rfcs/rfc3080.html What does this get us? * This system has mechanisms in place for streaming both a single large message and breaking a single reply up into multiple answers, allowing for pretty flexible streaming. (You can even mix these by having an answer that gets chunked itself.) * Concurrency is achieved by having multiple channels. Each channel executes messages in order, so you have a good mechanism for sending multiple things at once as well as maintaining ordering guarantees as necessary. * Reporting errors is very clear as it is a separate response type. * It has already been specified pretty clearly and we'd just be evolving that to something that more closely matches our needs. * It specifies sufficient data that you could implement this over transports other than TCP, such as UDP. Changes, rough list: * Use Avro-encoding for most things, so the encoding of a message would become an Avro struct. * Lose profiles in the sense that they're used in that specification since we're just exchanging Avro RPCs. * Do length prefixing rather than in the header, so that it is very amenable to binary I/O at high volumes. * No XML stuff, just existing things like the Avro handshake, wrapped up in messages. * For now, don't worry about things like flow control as expressed in RFC 3081, mapping of 3080 to TCP. * Think about adding something for true one-way messages, but an empty reply frame is probably sufficient, since that still allows reporting errors if needed (or desired). * May well need some extensions for a more flexible security model. * Use Avro RPC stuff to encode the channel management commands on channel 0 rather than XML. RFC 3117 (http://www.faqs.org/rfcs/rfc3117.html) goes into some of the philosophy and thinking behind the design of RFC 3080. Both are short and easy reading. - Bruce
Re: Thoughts on an RPC protocol
While I recommend actually reading RFC 3080 (it is an easy read), this
summary may help...
Framing: Length prefixed data, nothing unusual.
Encoding: Messages are effectively this:
enum message_type {
message,// a request
reply, // when there's only a single reply
answer, // when there are multiple replies, send multiple
answers and then a null.
null,// terminate a chain of replies
error, // oops, there was an error
}
struct message {
enum message_type message_type;
int channel;
int message_id;
bool more; // Is this message complete, or is more data coming?
for streaming
int sequence_number; // see RFC 3080
optional int answer_number; // Used for answers
bytes payload; // The actual RPC command, still serialized here
}
When a connection is opened, there's initially one channel, channel 0. That
channel is used for commands controlling the connection state, like opening
and closing channels. We should also perform Avro RPC handshakes over
channel 0.
Channels allow for concurrency. You can send requests/messages down
multiple channels and process them independently. Messages on a single
channel need to be processed in order though. This allows for both
guaranteed order of execution (within a single channel) and greater
concurrency (multiple channels).
Streaming happens in 2 ways.
The first way is to flip the more flag on a message. This means that the
data has been broken up over multiple messages and you need to receive the
whole thing before processing it.
The second is to have multiple answers (followed by a null frame) to a
single request message. This allows you to process the data in a streaming
fashion. The only thing that this doesn't allow is to process the data
being sent in a streaming fashion, but you could look at doing that by
sending multiple request messages instead.
Security and privacy can be handled by SASL.
The RFC defines a number of ways in which you can detect buggy
implementations of the protocol or invalid data being sent (framing /
encoding violations).
This should be pretty straight forward to implement, and as such (and since
I need such a thing in the immediate future), I've already begun an
implementation in C.
- Bruce
On Wed, Apr 7, 2010 at 4:13 PM, Bruce Mitchener
wrote:
> I'm assuming that the goals of an optimized transport for Avro RPC are
> something like the following:
>
> * Framing should be efficient, easy to implement.
> * Streaming of large values, both as part of a request and as a response
> is very important.
> * Being able to have multiple concurrent requests in flight, while also
> being able to have ordering guarantees where desired is necessary.
> * It should be easy to implement this in Java, C, Python, Ruby, etc.
> * Security is or will be important. This security can include
> authorization as well as privacy concerns.
>
> I'd like to see something based largely upon RFC 3080, with some
> simplifications and extensions:
>
> http://www.faqs.org/rfcs/rfc3080.html
>
> What does this get us?
>
> * This system has mechanisms in place for streaming both a single large
> message and breaking a single reply up into multiple answers, allowing for
> pretty flexible streaming. (You can even mix these by having an answer that
> gets chunked itself.)
> * Concurrency is achieved by having multiple channels. Each channel
> executes messages in order, so you have a good mechanism for sending
> multiple things at once as well as maintaining ordering guarantees as
> necessary.
> * Reporting errors is very clear as it is a separate response type.
> * It has already been specified pretty clearly and we'd just be evolving
> that to something that more closely matches our needs.
> * It specifies sufficient data that you could implement this over
> transports other than TCP, such as UDP.
>
> Changes, rough list:
>
> * Use Avro-encoding for most things, so the encoding of a message would
> become an Avro struct.
> * Lose profiles in the sense that they're used in that specification since
> we're just exchanging Avro RPCs.
> * Do length prefixing rather than in the header, so that it is very
> amenable to binary I/O at high volumes.
> * No XML stuff, just existing things like the Avro handshake, wrapped up
> in messages.
> * For now, don't worry about things like flow control as expressed in RFC
> 3081, mapping of 3080 to TCP.
> * Think about adding something for true one-way messages, but an empty
> reply frame is probably sufficient, since that still allows reporting errors
> if needed (or desired).
> * May well need some extensions for a more flexible security model.
> * Use Avro RPC stuff to encode the c
Re: Thoughts on an RPC protocol
Hi Bo,
Thanks for your feedback!
On Thu, Apr 8, 2010 at 3:49 PM, Bo Shi wrote:
> Hi Bruce,
>
> Would this RPC protocol take the role of the transport in the Avro
> specification or would it replace the protocol? If the handshake
> occurs on channel 0 while the request/response payloads are
> transferred on a different channel, this would not meet the existing
> wire protocol as described in the current 1.3.2 spec right?
>
This would be a transport.
I was thinking they would happen on 0 with 0 being a control channel, but
I'm not married to that. Offhand, I don't see why that would violate
anything in the specification though?
A couple other questions inline:
>
> On Thu, Apr 8, 2010 at 11:54 AM, Bruce Mitchener
> wrote:
> > While I recommend actually reading RFC 3080 (it is an easy read), this
> > summary may help...
> >
> > Framing: Length prefixed data, nothing unusual.
> > Encoding: Messages are effectively this:
> >
> > enum message_type {
> >message,// a request
> >reply, // when there's only a single reply
> >answer, // when there are multiple replies, send
> multiple
> > answers and then a null.
> >null,// terminate a chain of replies
> >error, // oops, there was an error
> > }
> >
> > struct message {
> >enum message_type message_type;
> >int channel;
> >int message_id;
> >bool more; // Is this message complete, or is more data
> coming?
> > for streaming
> >int sequence_number; // see RFC 3080
> >optional int answer_number; // Used for answers
> >bytes payload; // The actual RPC command, still serialized here
> > }
> >
> > When a connection is opened, there's initially one channel, channel 0.
> That
> > channel is used for commands controlling the connection state, like
> opening
> > and closing channels. We should also perform Avro RPC handshakes over
> > channel 0.
>
> Is channel 0 used exclusively as a control channel or would requests
> be allowed on this channel? Any idea on what the control messages
> would look like?
Channel 0 would be a control channel in my original proposal. I can see
arguments for allowing sending other requests on it from a simplicity point
of view. Thoughts?
Control messages would look like an Avro RPC call. They would be things
like:
OpenChannel returning the opened channel (or you could pass it the channel
number?)
CloseChannel gets passed the channel number to close.
> Channels allow for concurrency. You can send requests/messages down
> > multiple channels and process them independently. Messages on a single
> > channel need to be processed in order though. This allows for both
> > guaranteed order of execution (within a single channel) and greater
> > concurrency (multiple channels).
> >
> > Streaming happens in 2 ways.
>
> For streaming transfers, thoughts on optional compression codec
> attachment to streaming channels? It may be useful for IO-bound
> applications, but if you're transferring files like avro object
> container files that are already compressed - you'd need some extra
> coordination (but maybe that's outside the problem domain).
Compression would probably be something useful to support, agreed. That
could happen in various ways:
- Per channel (all data on the channel is compressed)
- Per request via a header of some sort
- Per connection (all data on all channels)
I suspect that I prefer per-channel from a simplicity point of view, but I'd
like to hear what people think.
- Bruce
Re: Thoughts on an RPC protocol
Doug, I'm happy to hear that you like this approach! Allocation of channels seems to be something specific to an application. In my app, I'd have a channel for the streaming data that is constantly arriving and a channel for making requests on and getting back answers immediately. Others could have a channel per object or whatever. Are your proxy servers custom software or are they just passing traffic along directly? If they're Avro-aware, then they can manage the handshaking process when routing to a new peer. Is this something that is actively happening today or just something that is possible? I definitely agree about not wanting a handshake per request. For my application that would add a lot of overhead in terms of the data transmitted. (I'm sending a lot of small requests, hopefully many thousands per second...) I would be much much happier being able to have a handshake per connection (or per channel open). - Bruce On Thu, Apr 8, 2010 at 4:43 PM, Doug Cutting wrote: > Bruce, > > Overall this looks like a good approach to me. > > How do you anticipate allocating channels? I'm guessing this would be one > per client object, that a pool of open connections to servers would be > maintained, and creating a new client object would allocate a new channel. > > Currently we perform a handshake per request. This is fairly cheap and > permits things like routing through proxy servers. Different requests over > the same connection can talk to different backend servers running different > versions of the protocol. Also consider the case where, between calls on an > object, the connection times out, and a new session is established and a new > handshake must take place. > > That said, having a session where the handshake can be assumed vastly > simplifies one-way messages. Without a response or error on which to prefix > a handshake response, a one-way client has no means to know that the server > was able to even parse its request. Yet we'd still like a handshake for > one-way messages, so that clients and servers need not be versioned in > lockstep. So the handshake-per-request model doesn't serve one-way messages > well. > > How can we address both of these needs: to permit flexible payload routing > and efficient one-way messaging? > > Doug > > > Bruce Mitchener wrote: > >> * Think about adding something for true one-way messages, but an empty >> reply frame is probably sufficient, since that still allows reporting >> errors >> if needed (or desired). >> >
Re: Thoughts on an RPC protocol
On Fri, Apr 9, 2010 at 10:00 AM, Scott Carey wrote: > > On Apr 8, 2010, at 11:35 PM, Bruce Mitchener wrote: > > > Doug, > > > > I'm happy to hear that you like this approach! > > > > Allocation of channels seems to be something specific to an application. > In > > my app, I'd have a channel for the streaming data that is constantly > > arriving and a channel for making requests on and getting back answers > > immediately. Others could have a channel per object or whatever. > > If this is all on one TCP port, then channels will interfere with one > another somewhat -- the transport layer will see packets arrive in the order > they were sent. If one packet in your streaming data stalls, both channels > will stall. Depending on the application requirements, this might be fine. > But it should be made clear that channels are not independent, they are > just interleaved over one ordered data stream. How each implementation > orders sending data on one end will affect order on the other side. Agreed, that's just a fact of life with TCP. Perhaps if SCTP ever gets some traction, then people can do a mapping for that. In the meantime, we could look at what RFC 3081 did in the TCP mapping for RFC 3080 with respect to flow control. > I definitely agree about not wanting a handshake per request. For my > > application that would add a lot of overhead in terms of the data > > transmitted. (I'm sending a lot of small requests, hopefully many > thousands > > per second...) I would be much much happier being able to have a > handshake > > per connection (or per channel open). > > > > Handshake per request will limit WAN usage. Doubling request latency isn't > a problem for local networks with sub 0.1ms RTTs, but it is a problem with > 25ms RTTs. Round trips aren't free on the processing or bandwidth side > either. If there is a way to meet most goals and limit extra handshakes to > specific cases that would be a significant performance improvement. We agree very strongly here. - Bruce
Re: Thoughts on an RPC protocol
What specific changes would you propose making to my proposal? - Bruce On Sat, Apr 10, 2010 at 11:57 AM, Jeff Hodges wrote: > Sorry for the spam. Python, java and apache httpd implementations > listed at the project page: http://www.chromium.org/spdy > > On Sat, Apr 10, 2010 at 10:53 AM, Jeff Hodges wrote: > > Oh, and it's been partially implemented in Chromium, so there's a > > quasi-reference implementation. > > -- > > Jeff > > > > On Sat, Apr 10, 2010 at 10:48 AM, Jeff Hodges > wrote: > >> To throw another set of ideas into the hat, SPDY[1][2] would be good > >> to learn from. SPDY takes the basics of HTTP and makes them fast. > >> Benefits we would enjoy include: > >> > >> * Multiplexed streams > >> * Request prioritization > >> * HTTP header compression > >> * Server push > >> > >> Currently in draft form. > >> > >> [1] http://dev.chromium.org/spdy/spdy-whitepaper > >> [2] http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2 > >> -- > >> Jeff > >> On Fri, Apr 9, 2010 at 2:29 PM, Doug Cutting > wrote: > >>> Scott Carey wrote: > > I also have not wrapped my head around routing/proxy use cases. From > a somewhat ignorant perspective on them -- I'd rather have a solid > point-to-point protocol that just works, is simple, and can meet the > vast majority of use cases with high performance than one that > happens to be capable of sophisticated routing but has a lot of other > limitations or is a lot harder to implement and debug. > >>> > >>> FWIW, they're theoretical at this point. I was only stating that > prefixing > >>> every request and response with handshakes makes stuff like proxies > trivial, > >>> since the protocol becomes stateless. Once we start having sessions > things > >>> get trickier. For example, many HTTP client libraries cache > connections, > >>> so, if you're building on top of one of those, it's hard to know when a > new > >>> connection is opened. > >>> > >>> One approach is to declare that the current framing and handshake rules > only > >>> apply to HTTP, currently our only standard transport. Then we can > define a > >>> new transport that's point-to-point, stateful, etc. which may handle > framing > >>> and handshakes differently. Thus we can retain back-compatibility. > Make > >>> sense? > >>> > >>> Doug > >>> > >> > > >
Re: Erlang Avro binding?
This was the last state of his work and as far as I know, no one else is working on it: http://github.com/toddlipcon/avro/tree/erl/src/erl/ It would be awesome to have Erlang support though! - Bruce On Wed, Apr 14, 2010 at 9:18 PM, satoshi kinoshita wrote: > > Hi, > > Does anyone there know anything about Erlang Avro binding? > I heard that Todd Lipcon started it last year but > I could not find any update or current states for Erlang binding. > > Thanks, > Satoshi Kinoshita >
Re: TLP move
+1 Sent from my iPhone On Apr 22, 2010, at 1:56 PM, Jeff Hammerbacher wrote: Switch to Confluence for the wiki, please. On Thu, Apr 22, 2010 at 10:19 AM, Doug Cutting wrote: The board yesterday passed a resolution making Avro a TLP. I filed an infrastructure issue to start the move: https://issues.apache.org/jira/browse/INFRA-2640 The primary disruption to developers will be when the subversion repository is renamed. We'll send out a note before we do this, then developers can use 'svn switch' to update their repos. One other issue is the wiki. I don't think it's easy to rename a subtree from a Moin Moin wiki to a new wiki. Fortunately we don't have many wiki pages and could cut and paste them manually. Alternately, we could switch to using confluence for our wiki. Thoughts? Doug
Re: new Avro committers: please update credits page
I tried to do the first part of this today (I am on OS X and didn't want to install everything for forrest yet). I got this error message: Sendingxdocs/credits.xml svn: Commit failed (details follow): svn: Server sent unexpected return value (403 Forbidden) in response to CHECKOUT request for '/repos/asf/!svn/ver/933317/hadoop/avro/site/author/content/xdocs/credits.xml' svn: Your commit message was left in a temporary file: svn: '/Users/bruce/Development/svn.apache/avro/site/author/content/svn-commit.tmp' I'm pretty sure that I used the right username and password on the command line with svn. Thoughts? - Bruce On Mon, Apr 12, 2010 at 11:02 PM, Doug Cutting wrote: > Can new committers please add themselves to the credits page? > > http://hadoop.apache.org/avro/credits.html > > This is done by checking out: > > https://svn.apache.org/repos/asf/hadoop/avro/site/ > > Edit author/content/xdocs/credits.xml, then: > > ant > # check contents of publish/credits.html in browser > svn commit -m "Updated credits page" > ssh people.apache.org > cd /www/hadoop.apache.org/avro > svn up > > Thanks, > > Doug > > >
[jira] Created: (AVRO-439) Remove unused headers from being checked in configure.in
Remove unused headers from being checked in configure.in Key: AVRO-439 URL: https://issues.apache.org/jira/browse/AVRO-439 Project: Avro Issue Type: Improvement Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener Priority: Minor The configure.in is checking for various headers explicitly ... but those headers are not used in the C implementation of Avro. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-439) Remove unused headers from being checked in configure.in
[ https://issues.apache.org/jira/browse/AVRO-439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-439: - Attachment: avro_configure.diff The attached patch removes the files checked for that aren't used. (Not that the checks for the other headers are used ... but that should be fixed separately.) > Remove unused headers from being checked in configure.in > > > Key: AVRO-439 > URL: https://issues.apache.org/jira/browse/AVRO-439 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Priority: Minor > Attachments: avro_configure.diff > > > The configure.in is checking for various headers explicitly ... but those > headers are not used in the C implementation of Avro. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-440) config.h output not correctly used
config.h output not correctly used -- Key: AVRO-440 URL: https://issues.apache.org/jira/browse/AVRO-440 Project: Avro Issue Type: Bug Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener While config.h is generated, it is only included from within st.c to make some things work correctly within st.h. I would suggest changing things a little: * Put an include of config.h into src/avro_private.h * Include avro_private.h into all .c files. * Not sure if the values from config.h are needed in any of the tests or examples ... I would hope not though and that this is fully insulated from being visible within anything exposed by avro.h. Given some feedback, I can readily prepare a patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-441) Support Universal binary builds on MacOS X
Support Universal binary builds on MacOS X -- Key: AVRO-441 URL: https://issues.apache.org/jira/browse/AVRO-441 Project: Avro Issue Type: Bug Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener Currently, MacOS X can't do universal builds because things are detected at configure time rather than compile time. I'll open an issue for each of these issues as i find them and link them to this one. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-442) sizeof void* and sizeof long detected at configure time
sizeof void* and sizeof long detected at configure time --- Key: AVRO-442 URL: https://issues.apache.org/jira/browse/AVRO-442 Project: Avro Issue Type: Sub-task Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener configure.in includes this: AC_CHECK_SIZEOF(void *) AC_CHECK_SIZEOF(long) Which is used in st.h: #if SIZEOF_LONG == SIZEOF_VOID_P typedef unsigned long st_data_t; #elif SIZEOF_LONG_LONG == SIZEOF_VOID_P typedef unsigned LONG_LONG st_data_t; #else #error >> st.c requires sizeof(void*) == sizeof(long) to be compiled. <<--- #endif #define ST_DATA_T_DEFINED Couldn't that just be uintptr_t or something else and avoid the configure check entirely? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-443) Endianness is determined at configure time rather than compile time
Endianness is determined at configure time rather than compile time --- Key: AVRO-443 URL: https://issues.apache.org/jira/browse/AVRO-443 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener configure.in includes: AC_C_BIGENDIAN That will define (or not) WORDS_BIGENDIAN ... which is used in src/encoding_binary.c That can be replaced with some compile-time checks ... I can submit a patch if necessary, let me know. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-441) Support Universal binary builds on MacOS X
[ https://issues.apache.org/jira/browse/AVRO-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840114#action_12840114 ] Bruce Mitchener commented on AVRO-441: -- The reason that this is important is that when doing a universal build the easier way (with -arch flags), the exact same code is compiled for each -arch flag on the gcc command line, so the source needs to be the same. > Support Universal binary builds on MacOS X > -- > > Key: AVRO-441 > URL: https://issues.apache.org/jira/browse/AVRO-441 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > > Currently, MacOS X can't do universal builds because things are detected at > configure time rather than compile time. > I'll open an issue for each of these issues as i find them and link them to > this one. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-439) Remove unused headers from being checked in configure.in
[ https://issues.apache.org/jira/browse/AVRO-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840115#action_12840115 ] Bruce Mitchener commented on AVRO-439: -- The same is also true for this: # Checks for library functions. AC_FUNC_MALLOC AC_CHECK_FUNCS([bzero]) (bzero isn't even used...) > Remove unused headers from being checked in configure.in > > > Key: AVRO-439 > URL: https://issues.apache.org/jira/browse/AVRO-439 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 >Reporter: Bruce Mitchener >Priority: Minor > Attachments: avro_configure.diff > > > The configure.in is checking for various headers explicitly ... but those > headers are not used in the C implementation of Avro. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-444) Fix warnings
Fix warnings Key: AVRO-444 URL: https://issues.apache.org/jira/browse/AVRO-444 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-444) Fix warnings
[ https://issues.apache.org/jira/browse/AVRO-444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-444: - Attachment: warnings.diff The attached patch fixes all warnings that I got on Mac OS X apart from unused parameter warnings. > Fix warnings > > > Key: AVRO-444 > URL: https://issues.apache.org/jira/browse/AVRO-444 > Project: Avro > Issue Type: Improvement > Components: c > Reporter: Bruce Mitchener > Attachments: warnings.diff > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-439) Remove unused headers from being checked in configure.in
[ https://issues.apache.org/jira/browse/AVRO-439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840195#action_12840195 ] Bruce Mitchener commented on AVRO-439: -- More up to date, since it cleans up a couple of other things and unistd.h can go away as well: http://github.com/waywardmonkeys/avro/commit/06c3ccab086de1f7025a7d7ee719b9054d05df42 > Remove unused headers from being checked in configure.in > > > Key: AVRO-439 > URL: https://issues.apache.org/jira/browse/AVRO-439 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Priority: Minor > Attachments: avro_configure.diff > > > The configure.in is checking for various headers explicitly ... but those > headers are not used in the C implementation of Avro. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-445) avro_size_data() to pre-calculate the size of an avro_datum_t in serialized form
avro_size_data() to pre-calculate the size of an avro_datum_t in serialized form Key: AVRO-445 URL: https://issues.apache.org/jira/browse/AVRO-445 Project: Avro Issue Type: New Feature Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener Attachments: avro_size.diff I would like to be able to find out how much buffer space will be required to serialize a given avro_datum_t ... I've started implementing this but wanted to get feedback on the general approach and API. The details of most of the functions in datum_size.c haven't been updated yet in the attached patch. The rest should be close to working though. The main point of concern that I have at the moment is that I'm returning an int64_t which, if positive, is the length required and if negative, will be an error condition (like -EINVAL). Thoughts? Anything else in the attached patch that should change? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-445) avro_size_data() to pre-calculate the size of an avro_datum_t in serialized form
[ https://issues.apache.org/jira/browse/AVRO-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-445: - Attachment: avro_size.diff Work in progress... > avro_size_data() to pre-calculate the size of an avro_datum_t in serialized > form > > > Key: AVRO-445 > URL: https://issues.apache.org/jira/browse/AVRO-445 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 >Reporter: Bruce Mitchener > Attachments: avro_size.diff > > > I would like to be able to find out how much buffer space will be required to > serialize a given avro_datum_t ... I've started implementing this but wanted > to get feedback on the general approach and API. > The details of most of the functions in datum_size.c haven't been updated yet > in the attached patch. The rest should be close to working though. > The main point of concern that I have at the moment is that I'm returning an > int64_t which, if positive, is the length required and if negative, will be > an error condition (like -EINVAL). > Thoughts? Anything else in the attached patch that should change? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-442) sizeof void* and sizeof long detected at configure time
[ https://issues.apache.org/jira/browse/AVRO-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-442: - Attachment: avro_no_size_checks.diff The attached patch just uses uintptr_t instead of doing configure checks... > sizeof void* and sizeof long detected at configure time > --- > > Key: AVRO-442 > URL: https://issues.apache.org/jira/browse/AVRO-442 > Project: Avro > Issue Type: Sub-task > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_no_size_checks.diff > > > configure.in includes this: > AC_CHECK_SIZEOF(void *) > AC_CHECK_SIZEOF(long) > Which is used in st.h: > #if SIZEOF_LONG == SIZEOF_VOID_P > typedef unsigned long st_data_t; > #elif SIZEOF_LONG_LONG == SIZEOF_VOID_P > typedef unsigned LONG_LONG st_data_t; > #else > #error >> st.c requires sizeof(void*) == sizeof(long) to be compiled. > <<--- > #endif > #define ST_DATA_T_DEFINED > Couldn't that just be uintptr_t or something else and avoid the configure > check entirely? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-439) Remove unused headers from being checked in configure.in
[ https://issues.apache.org/jira/browse/AVRO-439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-439: - Attachment: avro_configure.diff Updated, this is what I had on Github for this change. > Remove unused headers from being checked in configure.in > > > Key: AVRO-439 > URL: https://issues.apache.org/jira/browse/AVRO-439 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Priority: Minor > Attachments: avro_configure.diff, avro_configure.diff > > > The configure.in is checking for various headers explicitly ... but those > headers are not used in the C implementation of Avro. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-443) Endianness is determined at configure time rather than compile time
[ https://issues.apache.org/jira/browse/AVRO-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-443: - Attachment: avro_endian.diff The attached patch should resolve this. > Endianness is determined at configure time rather than compile time > --- > > Key: AVRO-443 > URL: https://issues.apache.org/jira/browse/AVRO-443 > Project: Avro > Issue Type: Sub-task > Components: c > Reporter: Bruce Mitchener > Attachments: avro_endian.diff > > > configure.in includes: > AC_C_BIGENDIAN > That will define (or not) WORDS_BIGENDIAN ... which is used in > src/encoding_binary.c > That can be replaced with some compile-time checks ... I can submit a patch > if necessary, let me know. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-448) encoding_binary.c doesn't build on little endian platforms
encoding_binary.c doesn't build on little endian platforms -- Key: AVRO-448 URL: https://issues.apache.org/jira/browse/AVRO-448 Project: Avro Issue Type: Bug Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener See upcoming attachment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-448) encoding_binary.c doesn't build on little endian platforms
[ https://issues.apache.org/jira/browse/AVRO-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-448: - Attachment: avro_little_endian.diff Simple fix attached. > encoding_binary.c doesn't build on little endian platforms > -- > > Key: AVRO-448 > URL: https://issues.apache.org/jira/browse/AVRO-448 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_little_endian.diff > > > See upcoming attachment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-445) avro_size_data() to pre-calculate the size of an avro_datum_t in serialized form
[ https://issues.apache.org/jira/browse/AVRO-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-445: - Attachment: avro_size.diff Attached patch appears to work and includes tests. > avro_size_data() to pre-calculate the size of an avro_datum_t in serialized > form > > > Key: AVRO-445 > URL: https://issues.apache.org/jira/browse/AVRO-445 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 >Reporter: Bruce Mitchener > Attachments: avro_size.diff, avro_size.diff > > > I would like to be able to find out how much buffer space will be required to > serialize a given avro_datum_t ... I've started implementing this but wanted > to get feedback on the general approach and API. > The details of most of the functions in datum_size.c haven't been updated yet > in the attached patch. The rest should be close to working though. > The main point of concern that I have at the moment is that I'm returning an > int64_t which, if positive, is the length required and if negative, will be > an error condition (like -EINVAL). > Thoughts? Anything else in the attached patch that should change? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-448) encoding_binary.c doesn't build on little endian platforms
[ https://issues.apache.org/jira/browse/AVRO-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840654#action_12840654 ] Bruce Mitchener commented on AVRO-448: -- I meant big endian platforms ... > encoding_binary.c doesn't build on little endian platforms > -- > > Key: AVRO-448 > URL: https://issues.apache.org/jira/browse/AVRO-448 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 >Reporter: Bruce Mitchener > Attachments: avro_little_endian.diff > > > See upcoming attachment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-445) avro_size_data() to pre-calculate the size of an avro_datum_t in serialized form
[ https://issues.apache.org/jira/browse/AVRO-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-445: - Attachment: (was: avro_size.diff) > avro_size_data() to pre-calculate the size of an avro_datum_t in serialized > form > > > Key: AVRO-445 > URL: https://issues.apache.org/jira/browse/AVRO-445 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 >Reporter: Bruce Mitchener > Attachments: avro_size.diff > > > I would like to be able to find out how much buffer space will be required to > serialize a given avro_datum_t ... I've started implementing this but wanted > to get feedback on the general approach and API. > The details of most of the functions in datum_size.c haven't been updated yet > in the attached patch. The rest should be close to working though. > The main point of concern that I have at the moment is that I'm returning an > int64_t which, if positive, is the length required and if negative, will be > an error condition (like -EINVAL). > Thoughts? Anything else in the attached patch that should change? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-448) encoding_binary.c doesn't build on little endian platforms
[ https://issues.apache.org/jira/browse/AVRO-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-448: - Attachment: (was: avro_little_endian.diff) > encoding_binary.c doesn't build on little endian platforms > -- > > Key: AVRO-448 > URL: https://issues.apache.org/jira/browse/AVRO-448 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_big_endian.diff > > > See upcoming attachment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-448) encoding_binary.c doesn't build on little endian platforms
[ https://issues.apache.org/jira/browse/AVRO-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-448: - Attachment: avro_big_endian.diff Re-attaching to grant license. > encoding_binary.c doesn't build on little endian platforms > -- > > Key: AVRO-448 > URL: https://issues.apache.org/jira/browse/AVRO-448 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_big_endian.diff > > > See upcoming attachment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-449) CMake-based build system for Avro/C
CMake-based build system for Avro/C --- Key: AVRO-449 URL: https://issues.apache.org/jira/browse/AVRO-449 Project: Avro Issue Type: New Feature Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener I am working on a CMake-based build system for Avro/C. This has these advantages over the current autoconf build system: * Easier to get features like universal binaries correct. * Works on Windows to make it easier to work on a Windows port of Avro/C. * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-449: - Attachment: avro_cmake.diff Work in progress Things left to do: * Documentation about usage. * Support both version numbers that the autotools build system is using. * Support for building documentation. * Support for the cscope target. * Finish up the pretty target (moving the ~ files). * Figure out what to do about config.h The attached patch just removes the only reference to config.h in the sources for now ... (src/st.c). This patch shouldn't go in yet! To use it, apply it ... and then in lang/c, do the following: * mkdir cbuild * cd cbuild * cmake .. * make But at this point, it builds, it can install, it passes the tests, it generates projects for various IDEs, etc. And on OS X, it will automatically build for PPC, x86 (i386) and x86-64 as a universal binary. > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841626#action_12841626 ] Bruce Mitchener commented on AVRO-449: -- It would not be that difficult. There is support for finding Boost and other stuff in CMake as well which should help out. It might be useful for whoever does that work to also have a look at the CMake files in this as this uses Boost and Doxygen: http://code.google.com/p/cpp-library-project-template/ > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-452) Include cleanup
Include cleanup --- Key: AVRO-452 URL: https://issues.apache.org/jira/browse/AVRO-452 Project: Avro Issue Type: Improvement Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener Attachments: avro_header_cleanup.diff This makes most of the changes that I suggested in AVRO-440, but it removes the last reference to config.h instead of putting it into avro_private.h It also adds a license block to a file that it was previously missing from. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-452) Include cleanup
[ https://issues.apache.org/jira/browse/AVRO-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-452: - Attachment: avro_header_cleanup.diff > Include cleanup > --- > > Key: AVRO-452 > URL: https://issues.apache.org/jira/browse/AVRO-452 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_header_cleanup.diff > > > This makes most of the changes that I suggested in AVRO-440, but it removes > the last reference to config.h instead of putting it into avro_private.h > It also adds a license block to a file that it was previously missing from. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-453) More warning cleanup
More warning cleanup Key: AVRO-453 URL: https://issues.apache.org/jira/browse/AVRO-453 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener Attachments: avro_warning_cleanup.diff This requires that the patch from AVRO-452 be applied. It fixes all remaining warnings when using the CMake build (and probably autotools). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-453) More warning cleanup
[ https://issues.apache.org/jira/browse/AVRO-453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-453: - Attachment: avro_warning_cleanup.diff > More warning cleanup > > > Key: AVRO-453 > URL: https://issues.apache.org/jira/browse/AVRO-453 > Project: Avro > Issue Type: Improvement > Components: c > Reporter: Bruce Mitchener > Attachments: avro_warning_cleanup.diff > > > This requires that the patch from AVRO-452 be applied. It fixes all remaining > warnings when using the CMake build (and probably autotools). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-441) Support Universal binary builds on MacOS X
[ https://issues.apache.org/jira/browse/AVRO-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841917#action_12841917 ] Bruce Mitchener commented on AVRO-441: -- This is working for me now if I use the CMake build. If we're in agreement that we're going to at least support CMake going forward, we can close this... If not, let me know and I'll try to find the time to look into what's wrong with the autotools build. > Support Universal binary builds on MacOS X > -- > > Key: AVRO-441 > URL: https://issues.apache.org/jira/browse/AVRO-441 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 >Reporter: Bruce Mitchener > > Currently, MacOS X can't do universal builds because things are detected at > configure time rather than compile time. > I'll open an issue for each of these issues as i find them and link them to > this one. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-440) config.h output not correctly used
[ https://issues.apache.org/jira/browse/AVRO-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-440: - Attachment: avro_config_h_check.diff Apply AVRO-452's patch and then this one ... it will allow the use of stuff via config.h in an autotools build without messing up a CMake build. That said, nothing actually uses config.h at this point. > config.h output not correctly used > -- > > Key: AVRO-440 > URL: https://issues.apache.org/jira/browse/AVRO-440 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_config_h_check.diff > > > While config.h is generated, it is only included from within st.c to make > some things work correctly within st.h. > I would suggest changing things a little: > * Put an include of config.h into src/avro_private.h > * Include avro_private.h into all .c files. > * Not sure if the values from config.h are needed in any of the tests or > examples ... I would hope not though and that this is fully insulated from > being visible within anything exposed by avro.h. > Given some feedback, I can readily prepare a patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-460) Performance improvement to write_long()
Performance improvement to write_long() --- Key: AVRO-460 URL: https://issues.apache.org/jira/browse/AVRO-460 Project: Avro Issue Type: Improvement Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener write_long should buffer up its output and write it at once to avoid overhead rather than writing byte-by-byte. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-460) Performance improvement to write_long()
[ https://issues.apache.org/jira/browse/AVRO-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-460: - Attachment: avro_write_long.diff This patch improves performance significantly for larger numbers (and doesn't hurt for smaller numbers). > Performance improvement to write_long() > --- > > Key: AVRO-460 > URL: https://issues.apache.org/jira/browse/AVRO-460 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener > Attachments: avro_write_long.diff > > > write_long should buffer up its output and write it at once to avoid overhead > rather than writing byte-by-byte. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-464) Some hash tables can be arrays instead.
Some hash tables can be arrays instead. --- Key: AVRO-464 URL: https://issues.apache.org/jira/browse/AVRO-464 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener As far as I can tell, there's no need for the integer->string hash tables in schemas or records ... they can be arrays instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-464) Some hash tables can be arrays instead.
[ https://issues.apache.org/jira/browse/AVRO-464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845193#action_12845193 ] Bruce Mitchener commented on AVRO-464: -- But right now, the code is iterating 0 to num entries so that still would not work. My thought was to progressively rework this from where it is to using an open addressed hash instead and then it can be more efficient and still support sparse entries in the future. I think we can get rid of a fair bit of overhead in the current implementation. I will have an experimental patch soon... > Some hash tables can be arrays instead. > --- > > Key: AVRO-464 > URL: https://issues.apache.org/jira/browse/AVRO-464 > Project: Avro > Issue Type: Improvement > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > > As far as I can tell, there's no need for the integer->string hash tables in > schemas or records ... they can be arrays instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-464) Rework internals of records and schemas for greater performance
[ https://issues.apache.org/jira/browse/AVRO-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-464: - Summary: Rework internals of records and schemas for greater performance (was: Some hash tables can be arrays instead.) I am broadening the scope of this. I have some larger changes underway and it will be easier as a single patch. The initial changes led to a change from 4.2 to 3.7 seconds for serializing the same record object 10,000,000 times without validation. I think we can do better though. > Rework internals of records and schemas for greater performance > --- > > Key: AVRO-464 > URL: https://issues.apache.org/jira/browse/AVRO-464 > Project: Avro > Issue Type: Improvement > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > > As far as I can tell, there's no need for the integer->string hash tables in > schemas or records ... they can be arrays instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-418) avro.h generates errors when included in C++ code
[ https://issues.apache.org/jira/browse/AVRO-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener reassigned AVRO-418: Assignee: Bruce Mitchener (was: Matt Massie) > avro.h generates errors when included in C++ code > - > > Key: AVRO-418 > URL: https://issues.apache.org/jira/browse/AVRO-418 > Project: Avro > Issue Type: Bug > Components: c >Reporter: Matt Massie > Assignee: Bruce Mitchener > Attachments: AVRO-418.patch > > > avro.h cannot be included in C++ code without errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-418) avro.h generates errors when included in C++ code
[ https://issues.apache.org/jira/browse/AVRO-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-418: - Attachment: avro_build_with_cpp.diff I think this is a better approach to addressing the problems ... it also includes a test program that will fail to compile if the other fixes aren't in place. I tested the test under the CMake build but NOT under autotools. > avro.h generates errors when included in C++ code > - > > Key: AVRO-418 > URL: https://issues.apache.org/jira/browse/AVRO-418 > Project: Avro > Issue Type: Bug > Components: c >Reporter: Matt Massie > Assignee: Bruce Mitchener > Attachments: AVRO-418.patch, avro_build_with_cpp.diff > > > avro.h cannot be included in C++ code without errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845524#action_12845524 ] Bruce Mitchener commented on AVRO-449: -- When you get a chance, please uncomment those lines (and leave the requirement at 2.4) and tell me how it is actually failing ... it seems like CMAKE_COMPILER_IS_GNUCC should have been working even back in 2.4... > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-449: - Attachment: (was: avro_cmake.diff) > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-449: - Attachment: avro_cmake.diff Updated to address the issues with CMake 2.4 on Linux. > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-462) C implementation does not accept "simple" primitive schemas
[
https://issues.apache.org/jira/browse/AVRO-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845705#action_12845705
]
Bruce Mitchener commented on AVRO-462:
--
The roadmap (http://wiki.github.com/akheron/jansson/roadmap) for Jansson 1.3
indicates that they want to support this ... wonder how hard it would be to
contribute the required support to Jansson vs changing to yajl?
> C implementation does not accept "simple" primitive schemas
> ---
>
> Key: AVRO-462
> URL: https://issues.apache.org/jira/browse/AVRO-462
> Project: Avro
> Issue Type: Bug
> Components: c
>Affects Versions: 1.3.0
>Reporter: Jeff Hodges
>
> Schemas like "null", "int", "string", etc. (instead of the longer, {type:
> "null"}, {type: "int"}, etc.) are not accepted by the C implementation.
> This seems to be a problem with the jansson JSON library used. It (strictly
> correctly) says that simple strings are not JSON objects.
> Switching to the yajl[1] library would fix this as yajl accepts "subsets" of
> JSON like strings, numbers and so on.
> [1] http://lloyd.github.com/yajl/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-465) C implementation requires you to know a file's schema before reading
[ https://issues.apache.org/jira/browse/AVRO-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845706#action_12845706 ] Bruce Mitchener commented on AVRO-465: -- I'd suggest this signature: avro_schema_t avro_file_reader_get_schema(avro_file_reader_t reader) > C implementation requires you to know a file's schema before reading > > > Key: AVRO-465 > URL: https://issues.apache.org/jira/browse/AVRO-465 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.0 >Reporter: Jeff Hodges > Attachments: AVRO-465-schema_for_reader.patch > > > The C implementation gives the user no way of reading the objects in a data > file without knowing the file's schema ahead of time. > While it does fill in the writers_schema part of the avro_file_reader_t on > read, this field is not available to the API as it is left out of avro.h. Two > options persent itself: 1) preserve the API as is and add a > avro_schema_from_file_reader() function or 2) move the avro_file_reader_t and > avro_file_writer_t structs to avro.h. > A third option, that I don't approve of, is providing a function that reads > from a datafile but uses the writers_schema in the reader given instead of > requiring another schema to be passed into it. This is problematic because > anyone using the API would have fewer debugging and testing options when > dealing with interop datasets. Any problem that occurs might just be the > schema in the file being off, or whatever. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-466) Add avro_t for global state and avro_start / avro_stop functions
Add avro_t for global state and avro_start / avro_stop functions Key: AVRO-466 URL: https://issues.apache.org/jira/browse/AVRO-466 Project: Avro Issue Type: Improvement Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener Assignee: Bruce Mitchener Patch coming shortly. This is a bit of a change since we'd now recommend that users always call this at startup / shutdown ... not sure of the best way to handle that for people that don't. I guess that'd make this a change for 1.4? My subsequent changes will depend on this (memory allocation routing and atom table). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-467) Finish up CMake build system missing features
Finish up CMake build system missing features - Key: AVRO-467 URL: https://issues.apache.org/jira/browse/AVRO-467 Project: Avro Issue Type: Improvement Components: c Affects Versions: 1.3.0 Reporter: Bruce Mitchener Placeholder bug to serve as a parent for all of the various remaining tasks for the CMake build system. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-468) Document usage of the CMake build system
Document usage of the CMake build system Key: AVRO-468 URL: https://issues.apache.org/jira/browse/AVRO-468 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener We need to document how to use the CMake-based build system, including some typical things like debug vs release vs release-with-debug-info builds. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-467) Finish up CMake build system missing features
[ https://issues.apache.org/jira/browse/AVRO-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener reassigned AVRO-467: Assignee: Bruce Mitchener > Finish up CMake build system missing features > - > > Key: AVRO-467 > URL: https://issues.apache.org/jira/browse/AVRO-467 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > > Placeholder bug to serve as a parent for all of the various remaining tasks > for the CMake build system. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-468) Document usage of the CMake build system
[ https://issues.apache.org/jira/browse/AVRO-468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener reassigned AVRO-468: Assignee: Bruce Mitchener > Document usage of the CMake build system > > > Key: AVRO-468 > URL: https://issues.apache.org/jira/browse/AVRO-468 > Project: Avro > Issue Type: Sub-task > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > > We need to document how to use the CMake-based build system, including some > typical things like debug vs release vs release-with-debug-info builds. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-468) CMake: Document usage of the CMake build system
[ https://issues.apache.org/jira/browse/AVRO-468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-468: - Labels: cmake (was: ) Summary: CMake: Document usage of the CMake build system (was: Document usage of the CMake build system) > CMake: Document usage of the CMake build system > --- > > Key: AVRO-468 > URL: https://issues.apache.org/jira/browse/AVRO-468 > Project: Avro > Issue Type: Sub-task > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > > We need to document how to use the CMake-based build system, including some > typical things like debug vs release vs release-with-debug-info builds. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-469) CMake: Support correct version numbers in the libraries
CMake: Support correct version numbers in the libraries --- Key: AVRO-469 URL: https://issues.apache.org/jira/browse/AVRO-469 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener We need to set both VERSION and SOVERSION correctly as the autotools build did. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-470) CMake: Support building documentation
CMake: Support building documentation - Key: AVRO-470 URL: https://issues.apache.org/jira/browse/AVRO-470 Project: Avro Issue Type: Sub-task Reporter: Bruce Mitchener Assignee: Bruce Mitchener We need to either support asciidoc with a build target or we need to switch to Doxygen and support it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-472) CMake: Complete the pretty target
CMake: Complete the pretty target - Key: AVRO-472 URL: https://issues.apache.org/jira/browse/AVRO-472 Project: Avro Issue Type: Sub-task Reporter: Bruce Mitchener Assignee: Bruce Mitchener I don't think I implemented support for moving the ~ files as the autotools target did. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-471) CMake: Support cscope target
CMake: Support cscope target Key: AVRO-471 URL: https://issues.apache.org/jira/browse/AVRO-471 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-473) CMake: Look at running the JSON / Jansson tests
CMake: Look at running the JSON / Jansson tests --- Key: AVRO-473 URL: https://issues.apache.org/jira/browse/AVRO-473 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener We could run the Jansson tests for the JSON code ... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-474) CMake: Add dist target
CMake: Add dist target -- Key: AVRO-474 URL: https://issues.apache.org/jira/browse/AVRO-474 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener We need a dist / distcheck target ... Might be worth looking at this which does it: http://bit.ly/d4emB8 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-475) CMake: What to do about config.h?
CMake: What to do about config.h? - Key: AVRO-475 URL: https://issues.apache.org/jira/browse/AVRO-475 Project: Avro Issue Type: Sub-task Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener We need to decide what to do about config.h ... autotools is still generating one but we don't actually use it anywhere. I think we can safely just get rid of it entirely for now until we have a demonstrated need for it? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-467) CMake: Complete CMake build system and remove autotools build system
[ https://issues.apache.org/jira/browse/AVRO-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-467: - Labels: cmake (was: ) Summary: CMake: Complete CMake build system and remove autotools build system (was: Finish up CMake build system missing features) > CMake: Complete CMake build system and remove autotools build system > > > Key: AVRO-467 > URL: https://issues.apache.org/jira/browse/AVRO-467 > Project: Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > > Placeholder bug to serve as a parent for all of the various remaining tasks > for the CMake build system. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-476) Documentation: Replace asciidoc with Doxygen?
Documentation: Replace asciidoc with Doxygen? - Key: AVRO-476 URL: https://issues.apache.org/jira/browse/AVRO-476 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener Doxygen is a lot more common, more likely to be installed and won't bring in a ton of extra dependencies to get it working... Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-470) CMake: Support building documentation
[ https://issues.apache.org/jira/browse/AVRO-470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-470: - Component/s: c > CMake: Support building documentation > - > > Key: AVRO-470 > URL: https://issues.apache.org/jira/browse/AVRO-470 > Project: Avro > Issue Type: Sub-task > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > > We need to either support asciidoc with a build target or we need to switch > to Doxygen and support it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-472) CMake: Complete the pretty target
[ https://issues.apache.org/jira/browse/AVRO-472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-472: - Component/s: c > CMake: Complete the pretty target > - > > Key: AVRO-472 > URL: https://issues.apache.org/jira/browse/AVRO-472 > Project: Avro > Issue Type: Sub-task > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > > I don't think I implemented support for moving the ~ files as the autotools > target did. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-477) Buildbots
Buildbots - Key: AVRO-477 URL: https://issues.apache.org/jira/browse/AVRO-477 Project: Avro Issue Type: Wish Components: c Reporter: Bruce Mitchener Assignee: Matt Massie I wish for buildbots! See http://ci.apache.org/buildbot.html for more information about what Apache provides. I would like to see one on each platform that they support building HEAD. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-451) Try to use hashlib in Python implementation and fall back to md5 if we can't find it
[ https://issues.apache.org/jira/browse/AVRO-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-451: - Attachment: avro_python_hashlib.diff Easy fix ... > Try to use hashlib in Python implementation and fall back to md5 if we can't > find it > > > Key: AVRO-451 > URL: https://issues.apache.org/jira/browse/AVRO-451 > Project: Avro > Issue Type: Improvement > Components: python >Affects Versions: 1.3.0 >Reporter: Jeff Hammerbacher > Attachments: avro_python_hashlib.diff > > > hashlib, new in 2.5, seems to be the better way to do things > (http://docs.python.org/library/hashlib.html). We want to support Python 2.4, > so the best way to handle this situation is to try to import hashlib and fall > back to md5 if we can't find it (similar to the json/simplejson dance). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-303) Regularly run Avro tests on little-endian and big-endian systems
[ https://issues.apache.org/jira/browse/AVRO-303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845769#action_12845769 ] Bruce Mitchener commented on AVRO-303: -- I've been talking with Massie about using Buildbot for Avro-C. This is in part due to the Buildbot infrastructure at Apache having broader platform support (FreeBSD, Solaris, a couple versions of Ubuntu, and Windows Server 2008 with Windows 7 on the way). I opened http://issues.apache.org/jira/browse/AVRO-477 to track this with Massie. > Regularly run Avro tests on little-endian and big-endian systems > > > Key: AVRO-303 > URL: https://issues.apache.org/jira/browse/AVRO-303 > Project: Avro > Issue Type: Test >Reporter: Jeff Hammerbacher > > To ensure that Avro implementations are working correctly, it would be nice > to have a shared Avro infrastructure for testing that contains both a > big-endian and a little-endian system. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-477) Buildbots
[ https://issues.apache.org/jira/browse/AVRO-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846039#action_12846039 ] Bruce Mitchener commented on AVRO-477: -- Does Hudson have the same OS coverage that Buildbot does? (Will we be able to test builds on various Unix and Windows versions?) If not, then we still need BuildBot. There's probably no harm in having both Hudson for those who want that and Buildbot for those who need the additional coverage. > Buildbots > - > > Key: AVRO-477 > URL: https://issues.apache.org/jira/browse/AVRO-477 > Project: Avro > Issue Type: Wish > Components: c > Reporter: Bruce Mitchener >Assignee: Matt Massie > > I wish for buildbots! > See http://ci.apache.org/buildbot.html for more information about what Apache > provides. > I would like to see one on each platform that they support building HEAD. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-477) Buildbots
[ https://issues.apache.org/jira/browse/AVRO-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846101#action_12846101 ] Bruce Mitchener commented on AVRO-477: -- I have filed https://issues.apache.org/jira/browse/INFRA-2548 to put in a request for the Buildbots after talking with Cutting on IRC. > Buildbots > - > > Key: AVRO-477 > URL: https://issues.apache.org/jira/browse/AVRO-477 > Project: Avro > Issue Type: Wish > Components: c > Reporter: Bruce Mitchener >Assignee: Matt Massie > > I wish for buildbots! > See http://ci.apache.org/buildbot.html for more information about what Apache > provides. > I would like to see one on each platform that they support building HEAD. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-477) Buildbots
[ https://issues.apache.org/jira/browse/AVRO-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-477: - Component/s: python > Buildbots > - > > Key: AVRO-477 > URL: https://issues.apache.org/jira/browse/AVRO-477 > Project: Avro > Issue Type: Wish > Components: c, python > Reporter: Bruce Mitchener >Assignee: Matt Massie > > I wish for buildbots! > See http://ci.apache.org/buildbot.html for more information about what Apache > provides. > I would like to see one on each platform that they support building HEAD. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-480) avro_flush() is in the header, but not implemented
avro_flush() is in the header, but not implemented -- Key: AVRO-480 URL: https://issues.apache.org/jira/browse/AVRO-480 Project: Avro Issue Type: Bug Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener avro_flush() is in the header, but not implemented. Instead, should ditch that and move the 3 functions from avro_private.h out into avro.h -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-418) avro.h generates errors when included in C++ code
[ https://issues.apache.org/jira/browse/AVRO-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-418: - Fix Version/s: 1.3.1 > avro.h generates errors when included in C++ code > - > > Key: AVRO-418 > URL: https://issues.apache.org/jira/browse/AVRO-418 > Project: Avro > Issue Type: Bug > Components: c >Reporter: Matt Massie > Assignee: Bruce Mitchener > Fix For: 1.3.1 > > Attachments: AVRO-418.patch, avro_build_with_cpp.diff > > > avro.h cannot be included in C++ code without errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-449: - Fix Version/s: 1.3.1 > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > Fix For: 1.3.1 > > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-480) avro_flush() is in the header, but not implemented
[ https://issues.apache.org/jira/browse/AVRO-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-480: - Attachment: avro_flush.diff > avro_flush() is in the header, but not implemented > -- > > Key: AVRO-480 > URL: https://issues.apache.org/jira/browse/AVRO-480 > Project: Avro > Issue Type: Bug > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > Attachments: avro_flush.diff > > > avro_flush() is in the header, but not implemented. > Instead, should ditch that and move the 3 functions from avro_private.h out > into avro.h -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-449) CMake-based build system for Avro/C
[ https://issues.apache.org/jira/browse/AVRO-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-449: - Fix Version/s: (was: 1.3.1) 1.3.2 > CMake-based build system for Avro/C > --- > > Key: AVRO-449 > URL: https://issues.apache.org/jira/browse/AVRO-449 > Project: Avro > Issue Type: New Feature > Components: c >Affects Versions: 1.3.0 > Reporter: Bruce Mitchener >Assignee: Bruce Mitchener > Fix For: 1.3.2 > > Attachments: avro_cmake.diff > > > I am working on a CMake-based build system for Avro/C. This has these > advantages over the current autoconf build system: > * Easier to get features like universal binaries correct. > * Works on Windows to make it easier to work on a Windows port of Avro/C. > * Lets us generate project files for Visual Studio and XCode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-418) avro.h generates errors when included in C++ code
[ https://issues.apache.org/jira/browse/AVRO-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-418: - Fix Version/s: (was: 1.3.1) 1.3.2 > avro.h generates errors when included in C++ code > - > > Key: AVRO-418 > URL: https://issues.apache.org/jira/browse/AVRO-418 > Project: Avro > Issue Type: Bug > Components: c >Reporter: Matt Massie > Assignee: Bruce Mitchener > Fix For: 1.3.2 > > Attachments: AVRO-418.patch, avro_build_with_cpp.diff > > > avro.h cannot be included in C++ code without errors -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-481) Buildbot warning fixes
Buildbot warning fixes -- Key: AVRO-481 URL: https://issues.apache.org/jira/browse/AVRO-481 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener Assignee: Bruce Mitchener Fix For: 1.3.2 Patching coming shortly... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-481) Buildbot warning fixes
[ https://issues.apache.org/jira/browse/AVRO-481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-481: - Attachment: avro_481.diff > Buildbot warning fixes > -- > > Key: AVRO-481 > URL: https://issues.apache.org/jira/browse/AVRO-481 > Project: Avro > Issue Type: Improvement > Components: c > Reporter: Bruce Mitchener > Assignee: Bruce Mitchener > Fix For: 1.3.2 > > Attachments: avro_481.diff > > > Patching coming shortly... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-477) Buildbots
[ https://issues.apache.org/jira/browse/AVRO-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847000#action_12847000 ] Bruce Mitchener commented on AVRO-477: -- We now have buildbot coverage on Ubuntu 9.10, Solaris and FreeBSD for C and Ubuntu 9.10 for Python ... I think we can close this now. > Buildbots > - > > Key: AVRO-477 > URL: https://issues.apache.org/jira/browse/AVRO-477 > Project: Avro > Issue Type: Wish > Components: c, python > Reporter: Bruce Mitchener >Assignee: Matt Massie > > I wish for buildbots! > See http://ci.apache.org/buildbot.html for more information about what Apache > provides. > I would like to see one on each platform that they support building HEAD. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (AVRO-485) JavaScript encoding/decoding
JavaScript encoding/decoding Key: AVRO-485 URL: https://issues.apache.org/jira/browse/AVRO-485 Project: Avro Issue Type: Wish Reporter: Bruce Mitchener It would be very nice to have Javascript encoding/decoding support. There are at least a couple of interested in streaming Avro-encoded data to a browser-based client. Working with binary data in JS is a bit ugly ... here's a sample: http://github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/bson/binary_parser.js -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-485) JavaScript encoding/decoding
[ https://issues.apache.org/jira/browse/AVRO-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848053#action_12848053 ] Bruce Mitchener commented on AVRO-485: -- I started this ... my work on it can be found here on my experimental I/O branch that I'm using for changes to Avro-C on Github: http://github.com/waywardmonkeys/avro/tree/experimental_io/lang/js/ Feel free to help out with this. > JavaScript encoding/decoding > > > Key: AVRO-485 > URL: https://issues.apache.org/jira/browse/AVRO-485 > Project: Avro > Issue Type: Wish >Reporter: Bruce Mitchener > > It would be very nice to have Javascript encoding/decoding support. There > are at least a couple of interested in streaming Avro-encoded data to a > browser-based client. > Working with binary data in JS is a bit ugly ... here's a sample: > http://github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/bson/binary_parser.js -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-285) request-only messages
[ https://issues.apache.org/jira/browse/AVRO-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848775#action_12848775 ] Bruce Mitchener commented on AVRO-285: -- I'd very much like to see this ... any chance? What needs to be done to bring this closer to fruition? > request-only messages > - > > Key: AVRO-285 > URL: https://issues.apache.org/jira/browse/AVRO-285 > Project: Avro > Issue Type: New Feature > Components: spec >Reporter: Doug Cutting > > It might be useful to have a standard mechanism in Avro for transmitting > messages that receive no response, not even an acknowledgement. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-492) python implementation should encode numbers as little-endian
[ https://issues.apache.org/jira/browse/AVRO-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850455#action_12850455 ] Bruce Mitchener commented on AVRO-492: -- STRUCT_INT and STRUCT_LONG are only used in the process of handling floats and doubles. The interop testing just tests that the file could be read ... it won't test that the data came through correctly currently. If there was an error in something like integer/long encoding/decoding, it would show up due to the other things that use that (like strings) that would fail were it wrong. The interop tests need to be modified so that at least some of the fields being serialized and read back in have the same value for each language and can be verified in each test that reads in the interop data. Make sense? > python implementation should encode numbers as little-endian > > > Key: AVRO-492 > URL: https://issues.apache.org/jira/browse/AVRO-492 > Project: Avro > Issue Type: Bug > Components: python >Affects Versions: 1.3.1 >Reporter: Jeff Hodges >Assignee: Jeff Hodges > Attachments: AVRO-492.patch > > > The current version of the Python implementation uses big-endian encoding > when writing and reading numeric data. The spec specifies that little-endian > encoding should be used instead[1]. > [1] > http://hadoop.apache.org/avro/docs/current/spec.html#binary_encode_primitive -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-230) Create a shared schema test directory structure
[ https://issues.apache.org/jira/browse/AVRO-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850458#action_12850458 ] Bruce Mitchener commented on AVRO-230: -- We need at least a simple version of this sooner rather than later (we aren't currently doing correct interop testing of the actual values of floats, doubles and such). We can have things without random data involved and a static directory of known values that we can write tests to work against, no? Keeping it simple would mean that someone could start knocking out some basic examples in an hour or two and then we'd just need a reader in each langauge to verify. > Create a shared schema test directory structure > --- > > Key: AVRO-230 > URL: https://issues.apache.org/jira/browse/AVRO-230 > Project: Avro > Issue Type: New Feature > Components: c, c++, java, python >Reporter: Matt Massie >Assignee: Matt Massie > > This is an example of my proposed directory structure: > * invalid_schemas/ > ** broken.json > ** wrong.json > * valid_data/ > ** foo_test/ > *** schema.json > *** json_data/ > valid_json_test_data.json > more_valid_json_test_data.json > *** binary_data/ > valid_binary_test_data.bin > more_test_data.bin > ** bar_test/ > *** schema.json > *** json_data/ > ... > * invalid_data/ > ** baz_test/ > *** schema.json > *** json_data/ > ... > *** binary_data/ > ... > This structure supports positive and negative tests for avro schemas, json > data and binary data. > * The "invalid_schema" directory holds a number of invalid schemas that > should fail to parse. > * The "valid_data" directory has a number of self-contained tests in separate > directories. Each test directory is required to have a "schema.json" file > that valid avro schema. The "json_data" and "binary_data" directories are > optional for each test. > * The "invalid_data" directory has the same rules as the "valid_data" > directory. The data files should fail during tests (negative testing). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-501) missing function in C api to access array elements after decoding an array.
[ https://issues.apache.org/jira/browse/AVRO-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener updated AVRO-501: - Attachment: avro_array_get.diff > missing function in C api to access array elements after decoding an array. > --- > > Key: AVRO-501 > URL: https://issues.apache.org/jira/browse/AVRO-501 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.2 >Reporter: Robert G. Jakabosky >Assignee: Bruce Mitchener > Attachments: avro_array_get.diff > > > The C Api has no function for accessing the element in an array datum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AVRO-501) missing function in C api to access array elements after decoding an array.
[ https://issues.apache.org/jira/browse/AVRO-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener reassigned AVRO-501: Assignee: Bruce Mitchener > missing function in C api to access array elements after decoding an array. > --- > > Key: AVRO-501 > URL: https://issues.apache.org/jira/browse/AVRO-501 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.2 >Reporter: Robert G. Jakabosky >Assignee: Bruce Mitchener > Attachments: avro_array_get.diff > > > The C Api has no function for accessing the element in an array datum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-501) missing function in C api to access array elements after decoding an array.
[ https://issues.apache.org/jira/browse/AVRO-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853136#action_12853136 ] Bruce Mitchener commented on AVRO-501: -- The attached patch is not sufficient to really address everything required ... more in another comment. > missing function in C api to access array elements after decoding an array. > --- > > Key: AVRO-501 > URL: https://issues.apache.org/jira/browse/AVRO-501 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.2 >Reporter: Robert G. Jakabosky >Assignee: Bruce Mitchener > Attachments: avro_array_get.diff > > > The C Api has no function for accessing the element in an array datum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-501) missing function in C api to access array elements after decoding an array.
[ https://issues.apache.org/jira/browse/AVRO-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853263#action_12853263 ] Bruce Mitchener commented on AVRO-501: -- This patch is not sufficient ... We also need to provide read access to the length ... And similar stuff for unions, we need to provide read access to the discriminant and the value. I'd talked about this some with the reporter, but haven't gotten around to doing the patch yet. > missing function in C api to access array elements after decoding an array. > --- > > Key: AVRO-501 > URL: https://issues.apache.org/jira/browse/AVRO-501 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.2 >Reporter: Robert G. Jakabosky >Assignee: Bruce Mitchener > Attachments: avro_array_get.diff > > > The C Api has no function for accessing the element in an array datum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (AVRO-501) missing function in C api to access array elements after decoding an array.
[ https://issues.apache.org/jira/browse/AVRO-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Mitchener reopened AVRO-501: -- As per my last comment... > missing function in C api to access array elements after decoding an array. > --- > > Key: AVRO-501 > URL: https://issues.apache.org/jira/browse/AVRO-501 > Project: Avro > Issue Type: Bug > Components: c >Affects Versions: 1.3.2 >Reporter: Robert G. Jakabosky >Assignee: Bruce Mitchener > Attachments: avro_array_get.diff > > > The C Api has no function for accessing the element in an array datum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
