[jira] [Updated] (AVRO-1946) If user exists by default in docker, there will be an error, and the setup fails.
[ https://issues.apache.org/jira/browse/AVRO-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya updated AVRO-1946: Attachment: AVRO-1946.patch Initial patch. > If user exists by default in docker, there will be an error, and the setup > fails. > - > > Key: AVRO-1946 > URL: https://issues.apache.org/jira/browse/AVRO-1946 > Project: Avro > Issue Type: Bug > Components: docker >Affects Versions: 1.9.0 >Reporter: Suraj Acharya >Assignee: Suraj Acharya >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1946.patch > > > Currently, when a user already exists in a docker, it errors and stops > running. > However, for a group it continues after throwing a warning. This is > inconsistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1946) If user exists by default in docker, there will be an error, and the setup fails.
[ https://issues.apache.org/jira/browse/AVRO-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Acharya updated AVRO-1946: Status: Patch Available (was: Open) > If user exists by default in docker, there will be an error, and the setup > fails. > - > > Key: AVRO-1946 > URL: https://issues.apache.org/jira/browse/AVRO-1946 > Project: Avro > Issue Type: Bug > Components: docker >Affects Versions: 1.9.0 >Reporter: Suraj Acharya >Assignee: Suraj Acharya >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1946.patch > > > Currently, when a user already exists in a docker, it errors and stops > running. > However, for a group it continues after throwing a warning. This is > inconsistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AVRO-1946) If user exists by default in docker, there will be an error, and the setup fails.
Suraj Acharya created AVRO-1946: --- Summary: If user exists by default in docker, there will be an error, and the setup fails. Key: AVRO-1946 URL: https://issues.apache.org/jira/browse/AVRO-1946 Project: Avro Issue Type: Bug Components: docker Affects Versions: 1.9.0 Reporter: Suraj Acharya Assignee: Suraj Acharya Priority: Minor Fix For: 1.9.0 Currently, when a user already exists in a docker, it errors and stops running. However, for a group it continues after throwing a warning. This is inconsistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1827) Handling correctly optional fields when converting Protobuf to Avro
[ https://issues.apache.org/jira/browse/AVRO-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629416#comment-15629416 ] Karel Fuka commented on AVRO-1827: -- I had to ask Jakub to attach the patch as I do not have the rights to do so. It is in now. > Handling correctly optional fields when converting Protobuf to Avro > --- > > Key: AVRO-1827 > URL: https://issues.apache.org/jira/browse/AVRO-1827 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.7.7, 1.8.0 >Reporter: Jakub Kahovec > Attachments: AVRO-1827.patch, AVRO-1827.patch, AVRO-1827.patch, > AVRO-1827.patch > > > Hello, > as of the current implementation of converting protobuf files into avro > format, protobuf optional fields are being given default values in the avro > schema if not specified explicitly. > So for instance when the protobuf field is defined as > {quote} > optional int64 fieldInt64 = 1; > {quote} > in the avro schema it appears as > {quote} > "name" : "fieldInt64", > "type" : "long", > "default" : 0 > {quote} > The problem with this implementation is that we are losing information about > whether the field was present or not in the original protobuf, as when we ask > for this field's value in avro we will be given the default value. > What I'm proposing instead is that if the field in the protobuf is defined as > optional and has no default value then the generated avro schema type will us > a union comprising the matching type and null type with default value null. > It is going to look like this: > {quote} > "name" : "fieldIn64", > "type" : [ "null", "long" ], > "default" : null > {quote} > I'm aware that is a breaking change but I think that is the proper way how to > handle optional fields. > I've also created a patch which fixes the conversion > Jakub -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1827) Handling correctly optional fields when converting Protobuf to Avro
[ https://issues.apache.org/jira/browse/AVRO-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629421#comment-15629421 ] Karel Fuka commented on AVRO-1827: -- Thanks Michal. The latest patch I have attached is in Unix convention, so to apply it is easy. Just go to a directory where your patch is and apply it: {code} (git clone https://github.com/apache/avro.git; cd avro; patch -p0 < ../AVRO-1827.patch) {code} > Handling correctly optional fields when converting Protobuf to Avro > --- > > Key: AVRO-1827 > URL: https://issues.apache.org/jira/browse/AVRO-1827 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.7.7, 1.8.0 >Reporter: Jakub Kahovec > Attachments: AVRO-1827.patch, AVRO-1827.patch, AVRO-1827.patch, > AVRO-1827.patch > > > Hello, > as of the current implementation of converting protobuf files into avro > format, protobuf optional fields are being given default values in the avro > schema if not specified explicitly. > So for instance when the protobuf field is defined as > {quote} > optional int64 fieldInt64 = 1; > {quote} > in the avro schema it appears as > {quote} > "name" : "fieldInt64", > "type" : "long", > "default" : 0 > {quote} > The problem with this implementation is that we are losing information about > whether the field was present or not in the original protobuf, as when we ask > for this field's value in avro we will be given the default value. > What I'm proposing instead is that if the field in the protobuf is defined as > optional and has no default value then the generated avro schema type will us > a union comprising the matching type and null type with default value null. > It is going to look like this: > {quote} > "name" : "fieldIn64", > "type" : [ "null", "long" ], > "default" : null > {quote} > I'm aware that is a breaking change but I think that is the proper way how to > handle optional fields. > I've also created a patch which fixes the conversion > Jakub -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AVRO-1827) Handling correctly optional fields when converting Protobuf to Avro
[ https://issues.apache.org/jira/browse/AVRO-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakub Kahovec updated AVRO-1827: Attachment: AVRO-1827.patch > Handling correctly optional fields when converting Protobuf to Avro > --- > > Key: AVRO-1827 > URL: https://issues.apache.org/jira/browse/AVRO-1827 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.7.7, 1.8.0 >Reporter: Jakub Kahovec > Attachments: AVRO-1827.patch, AVRO-1827.patch, AVRO-1827.patch, > AVRO-1827.patch > > > Hello, > as of the current implementation of converting protobuf files into avro > format, protobuf optional fields are being given default values in the avro > schema if not specified explicitly. > So for instance when the protobuf field is defined as > {quote} > optional int64 fieldInt64 = 1; > {quote} > in the avro schema it appears as > {quote} > "name" : "fieldInt64", > "type" : "long", > "default" : 0 > {quote} > The problem with this implementation is that we are losing information about > whether the field was present or not in the original protobuf, as when we ask > for this field's value in avro we will be given the default value. > What I'm proposing instead is that if the field in the protobuf is defined as > optional and has no default value then the generated avro schema type will us > a union comprising the matching type and null type with default value null. > It is going to look like this: > {quote} > "name" : "fieldIn64", > "type" : [ "null", "long" ], > "default" : null > {quote} > I'm aware that is a breaking change but I think that is the proper way how to > handle optional fields. > I've also created a patch which fixes the conversion > Jakub -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1941) avrogencpp: unlike java, c++ generated code doesn't include own schema
[ https://issues.apache.org/jira/browse/AVRO-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629311#comment-15629311 ] ASF GitHub Bot commented on AVRO-1941: -- GitHub user moriarty reopened a pull request: https://github.com/apache/avro/pull/147 AVRO-1941: C++: adds __SCHEMA__ in generated code This adds an `avro::ValidSchema __SCHEMA__` to each generated struct. `__SCHEMA__` is initialized in constructor using `avro::compileJsonSchemaFromString`. This is a proposal for convenient way to access a schema similar to the generated Java classes, which have: ``` public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse(...); public static org.apache.avro.Schema getClassSchema() {return SCHEMA$; } ``` This is a proposal and doesn't add `__SCHEMA__` to enums, which the Java generated classes do have. I couldn't think of a c++ way to do that, and am still waiting on feedback for [AVRO-1940](https://issues.apache.org/jira/browse/AVRO-1940) or GH apache/avro#146 You can merge this pull request into a Git repository by running: $ git pull https://github.com/moriarty/avro avro-1941/cpp-include-schema-in-generated-structs/dev Alternatively you can review and apply these changes as the patch at: https://github.com/apache/avro/pull/147.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #147 commit 945b60106b1f775afceb5006575c52e9af893f97 Author: Alexander MoriartyDate: 2016-10-25T11:50:49Z AVRO-1941: C++: adds __SCHEMA__ in generated code This adds an avro::ValidSchema __SCHEMA__ to each generated struct. __SCHEMA__ is initialized in constructor using avro::compileJsonSchemaFromString. This is a proposal for convenient way to access a schema similar to the generated Java classes, which have: ``` public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse(...); public static org.apache.avro.Schema getClassSchema() {return SCHEMA$; } ``` This is a proposal and doesn't add __SCHEMA__ to enums, which the Java generated classes do have. I couldn't think of a c++ way to do that, and am still waiting on feedback for AVRO-1940 or GH apache/avro#146 > avrogencpp: unlike java, c++ generated code doesn't include own schema > -- > > Key: AVRO-1941 > URL: https://issues.apache.org/jira/browse/AVRO-1941 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.1 >Reporter: Alexander Moriarty > > All the classes generated for java include a copy of their schema. The C++ > classes do not. > Every generated java class has getClassSchema(). > Something similar for C++ would be convenient. > During generation, the schema is available with n->printJson(..) which can > then be passed through boost::algorithm::erase_all() to remove whitespace and > new lines, and then boost::algorithm::replace_all() to add escape characters > back. > and then given to avro::compileJsonSchemaFromString -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] avro pull request #147: AVRO-1941: C++: adds __SCHEMA__ in generated code
GitHub user moriarty reopened a pull request: https://github.com/apache/avro/pull/147 AVRO-1941: C++: adds __SCHEMA__ in generated code This adds an `avro::ValidSchema __SCHEMA__` to each generated struct. `__SCHEMA__` is initialized in constructor using `avro::compileJsonSchemaFromString`. This is a proposal for convenient way to access a schema similar to the generated Java classes, which have: ``` public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse(...); public static org.apache.avro.Schema getClassSchema() {return SCHEMA$; } ``` This is a proposal and doesn't add `__SCHEMA__` to enums, which the Java generated classes do have. I couldn't think of a c++ way to do that, and am still waiting on feedback for [AVRO-1940](https://issues.apache.org/jira/browse/AVRO-1940) or GH apache/avro#146 You can merge this pull request into a Git repository by running: $ git pull https://github.com/moriarty/avro avro-1941/cpp-include-schema-in-generated-structs/dev Alternatively you can review and apply these changes as the patch at: https://github.com/apache/avro/pull/147.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #147 commit 945b60106b1f775afceb5006575c52e9af893f97 Author: Alexander MoriartyDate: 2016-10-25T11:50:49Z AVRO-1941: C++: adds __SCHEMA__ in generated code This adds an avro::ValidSchema __SCHEMA__ to each generated struct. __SCHEMA__ is initialized in constructor using avro::compileJsonSchemaFromString. This is a proposal for convenient way to access a schema similar to the generated Java classes, which have: ``` public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse(...); public static org.apache.avro.Schema getClassSchema() {return SCHEMA$; } ``` This is a proposal and doesn't add __SCHEMA__ to enums, which the Java generated classes do have. I couldn't think of a c++ way to do that, and am still waiting on feedback for AVRO-1940 or GH apache/avro#146 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (AVRO-1827) Handling correctly optional fields when converting Protobuf to Avro
[ https://issues.apache.org/jira/browse/AVRO-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629209#comment-15629209 ] Karel Fuka commented on AVRO-1827: -- Hi I had been working on the original patch with Jakub and because this is becoming critical for us I am moving forward with it. [~tinxu] - thanks for your valuable feedback and you are right, there was an issue with the original patch. Though, it is a bit more subtle not completely related to a nested union. Let me explain: 1. If I run the following, it is all fine: {code:title=sample} public static enum E { A, B }; public void test1() { Schema s3 = Schema.createUnion(Arrays.asList( Schema.create(Schema.Type.NULL), ReflectData.get().getSchema(E.class))); int ind = ReflectData.get().resolveUnion(s3, E.A); } {code} 2. When I generate a ProtoBuf with the following definition and try to convert it to Avro, I really experience an exception thrown by resolveUnion(): {code:title=proto} message EMessage { enum ET { A = 1; B = 2; } optional ET et = 3; } {code} Note that the schema for 'et' is equivalent to what you see above. The real problem lies in 'ProtobufData' class, which extends 'GenericData' but does not override some methods (like 'getEnumSchema'), so a GenericData implementation gets called instead (and fails). Note that for example 'getEnumSchema' comment contains a text reading _"May be overridden for alternate enum"_. Therefore, I have addressed this in the latest version, which I am going to attach. Please review and let me know if you see any more issues. Thanks Karel > Handling correctly optional fields when converting Protobuf to Avro > --- > > Key: AVRO-1827 > URL: https://issues.apache.org/jira/browse/AVRO-1827 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.7.7, 1.8.0 >Reporter: Jakub Kahovec > Attachments: AVRO-1827.patch, AVRO-1827.patch, AVRO-1827.patch > > > Hello, > as of the current implementation of converting protobuf files into avro > format, protobuf optional fields are being given default values in the avro > schema if not specified explicitly. > So for instance when the protobuf field is defined as > {quote} > optional int64 fieldInt64 = 1; > {quote} > in the avro schema it appears as > {quote} > "name" : "fieldInt64", > "type" : "long", > "default" : 0 > {quote} > The problem with this implementation is that we are losing information about > whether the field was present or not in the original protobuf, as when we ask > for this field's value in avro we will be given the default value. > What I'm proposing instead is that if the field in the protobuf is defined as > optional and has no default value then the generated avro schema type will us > a union comprising the matching type and null type with default value null. > It is going to look like this: > {quote} > "name" : "fieldIn64", > "type" : [ "null", "long" ], > "default" : null > {quote} > I'm aware that is a breaking change but I think that is the proper way how to > handle optional fields. > I've also created a patch which fixes the conversion > Jakub -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] avro pull request #147: AVRO-1941: C++: adds __SCHEMA__ in generated code
Github user moriarty closed the pull request at: https://github.com/apache/avro/pull/147 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (AVRO-1941) avrogencpp: unlike java, c++ generated code doesn't include own schema
[ https://issues.apache.org/jira/browse/AVRO-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629077#comment-15629077 ] ASF GitHub Bot commented on AVRO-1941: -- Github user moriarty closed the pull request at: https://github.com/apache/avro/pull/147 > avrogencpp: unlike java, c++ generated code doesn't include own schema > -- > > Key: AVRO-1941 > URL: https://issues.apache.org/jira/browse/AVRO-1941 > Project: Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.8.1 >Reporter: Alexander Moriarty > > All the classes generated for java include a copy of their schema. The C++ > classes do not. > Every generated java class has getClassSchema(). > Something similar for C++ would be convenient. > During generation, the schema is available with n->printJson(..) which can > then be passed through boost::algorithm::erase_all() to remove whitespace and > new lines, and then boost::algorithm::replace_all() to add escape characters > back. > and then given to avro::compileJsonSchemaFromString -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1605) Remove Jackson classes from public API
[ https://issues.apache.org/jira/browse/AVRO-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628478#comment-15628478 ] Gabor Szadovszky commented on AVRO-1605: - {{Accessor.defaultValue(Field)}} is used to invoke {{ResolvingGrammarGenerator.encode(Encoder, Schema, JsonNode)}}. The implementation of that method uses the jackson API. It might be refactored to work on the common objects; I just didn't want to introduce regressions here and let the patch grow too large. - {{Accessor.createField}}, {{Accessor.addProp}}: These are mainly used by the generated class {{Idl}}. This class extensively uses the Jackson API and build structures from these objects. It seems to be a hard work to refactor this class to not to use these objects. Meanwhile, I was thinking about the concept of this friend package pattern. I am not sure I share you opinion that it should be avoided. For me it is more solving a problem that the java language should. Package private visibility is simply not enough to handle such issues. It might be a decision though, to not to use a lib like jackson such an extensive way but in this case it is not enough to remove those classes from the public methods. We should not use JsonNode objects even privately but use jackson only for parsing and writing jsons. In the current situation I am not sure it would worth the effort. I'm sure I see things differently than you as I do not have too much xp in the avro code base. Therefore, I am really curious about your opinion. :) > Remove Jackson classes from public API > -- > > Key: AVRO-1605 > URL: https://issues.apache.org/jira/browse/AVRO-1605 > Project: Avro > Issue Type: Sub-task > Components: java >Affects Versions: 1.7.8 >Reporter: Tom White >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1943) Flaky test: TestNettyServerWithCompression.testConnectionsCount
[ https://issues.apache.org/jira/browse/AVRO-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628330#comment-15628330 ] ASF GitHub Bot commented on AVRO-1943: -- Github user gszadovszky closed the pull request at: https://github.com/apache/avro/pull/148 > Flaky test: TestNettyServerWithCompression.testConnectionsCount > --- > > Key: AVRO-1943 > URL: https://issues.apache.org/jira/browse/AVRO-1943 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Gabor Szadovszky >Assignee: Gabor Szadovszky > Fix For: 1.9.0 > > > One of the unit tests is often failing in our jenkins environment. However, I > was not able to reproduce it on my own environment; it might be related to > either the CPU or the network load. > Stack Trace: > {code} > junit.framework.AssertionFailedError: expected:<1> but was:<2> > at junit.framework.Assert.fail(Assert.java:50) > at junit.framework.Assert.failNotEquals(Assert.java:287) > at junit.framework.Assert.assertEquals(Assert.java:67) > at junit.framework.Assert.assertEquals(Assert.java:199) > at junit.framework.Assert.assertEquals(Assert.java:205) > at > org.apache.avro.ipc.TestNettyServer.testConnectionsCount(TestNettyServer.java:152) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] avro pull request #148: AVRO-1943 - Flaky test: TestNettyServerWithCompressi...
Github user gszadovszky closed the pull request at: https://github.com/apache/avro/pull/148 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---