Re: Contributor Guide > Python Unit Testing Instructions

[Michael A. Smith]

Hi, Sandeep,

If you want to run just the unit tests for python, you can use tox. Just
running `tox` by itself should be sufficient if you have at least one of
the supported versions of python installed.

I'll take a look at getting those docs updated.

On Wed, Apr 22, 2020 at 19:50 Sandeep Narayanaswami
 wrote:

> Hi Avro devs,
>
> The How To Contribute
> 
> Confluence
> page says to run unit tests in lang/py using ant. The work on AVRO-831
> seems to have changed the directory structure such that build.xml no longer
> addresses the correct folders. Is ant still the preferred means of running
> unit tests, or is this deprecated in favor of a setup.py approach?
> I have tried setup.py build test in lang/py, and the tests run, albeit with
> a handful of failures due to missing libraries. If this is the way to go,
> any advice on resolving the failures?
>
> Cheers,
> Sandeep
>
> __
>
>
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
>
>

[GitHub] [avro] anhldbk commented on a change in pull request #842: AVRO-2723: Refactor ReflectData to allow derived classes to customize default values for fields

[GitBox]

anhldbk commented on a change in pull request #842:
URL: https://github.com/apache/avro/pull/842#discussion_r413445873



##
File path: lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
##
@@ -110,14 +112,96 @@ public static ReflectData get() {
   }
 
   /**
-   * Cause a class to be treated as though it had an {@link Stringable}
-   ** annotation.
+   * Cause a class to be treated as though it had an {@link Stringable} *
+   * annotation.
*/
   public ReflectData addStringable(Class c) {
 stringableClasses.add(c);
 return this;
   }
 
+  /**
+   * If this flag is set to true, default values for fields will be assigned
+   * dynamically using Java reflections. Let's call this feature `default
+   * reflection`. Initially this feature is disabled
+   */
+  private boolean defaultGenerated = false;
+
+  /**
+   * Enable or disable `default reflection`
+   *
+   * @param enabled set to `true` to enable the feature. This feature is 
disabled
+   *by default
+   * @return The current instance
+   */
+  public ReflectData setDefaultsGenerated(boolean enabled) {
+this.defaultGenerated = enabled;
+return this;
+  }
+
+  private final Map defaultValues = new WeakHashMap<>();

Review comment:
   @cutting Agree. One small question: can I use `Map` 
instead?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] kojiromike commented on a change in pull request #847: Avro-2785: Update specs on how unions encoded

[GitBox]

kojiromike commented on a change in pull request #847:
URL: https://github.com/apache/avro/pull/847#discussion_r413410884



##
File path: doc/src/content/xdocs/spec.xml
##
@@ -546,7 +546,7 @@
 
   
 Unions
-A union is encoded by first writing a long
+A union is encoded by first writing an int

Review comment:
   Until the needed consideration for c/++ are done, I think we should 
include a parenthetical for that case. Something like _(this can theoretically 
be a long, and is allowed as such in C and C++, but we don't anticipate unions 
with 215M members)_.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AVRO-2785) Update specs on how unions encoded

[Michael A. Smith (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090093#comment-17090093
 ] 

Michael A. Smith commented on AVRO-2785:


Hi, [~anhldbk], do you want to open a sibling ticket to this one to track the 
changes needed in C/++?

> Update specs on how unions encoded
> --
>
> Key: AVRO-2785
> URL: https://issues.apache.org/jira/browse/AVRO-2785
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: doc, spec
>Affects Versions: 1.9.2
>Reporter: Andy Le
>Priority: Major
>
> h1. Overview
> This issue is associated with [our mailing 
> discussions|https://lists.apache.org/thread.html/r3f4b8b40ec9604e4f3854fb59f9219db9bbb58eebea4b9d2aa097688%40%3Cuser.avro.apache.org%3E]
> h1. Solution
> Update the Spec



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Contributor Guide > Python Unit Testing Instructions

[Sandeep Narayanaswami]

Hi Avro devs,

The How To Contribute
 Confluence
page says to run unit tests in lang/py using ant. The work on AVRO-831
seems to have changed the directory structure such that build.xml no longer
addresses the correct folders. Is ant still the preferred means of running
unit tests, or is this deprecated in favor of a setup.py approach?
I have tried setup.py build test in lang/py, and the tests run, albeit with
a handful of failures due to missing libraries. If this is the way to go,
any advice on resolving the failures?

Cheers,
Sandeep

__



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

[jira] [Created] (AVRO-2812) Avro maven plugin

[Amsterdam Luis de Lima Filho (Jira)]

Amsterdam Luis de Lima Filho created AVRO-2812:
--

 Summary: Avro maven plugin
 Key: AVRO-2812
 URL: https://issues.apache.org/jira/browse/AVRO-2812
 Project: Apache Avro
  Issue Type: Improvement
  Components: java
Reporter: Amsterdam Luis de Lima Filho


Currently, the Avro Maven Plugin offer two options related to Optionals: 
gettersReturnOptional and createOptionalGetters. But the more useful option is 
missing, which is to create just the optional getter when the field accepts 
null, and to create just the normal getter when the field does not accept null. 
There is no workaround for this, so this feature would be very desirable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [avro] kojiromike opened a new pull request #861: AVRO-2748: Refactor Schema Matching

[GitBox]

kojiromike opened a new pull request #861:
URL: https://github.com/apache/avro/pull/861


   Leverage each schema type's internal knowledge of its matching logic to
   simplify schema matching code. This object-oriented refactor is intended
   to make it easier to approach tickets like AVRO-2748. It doesn't
   actually resolve the issue, but makes it possible to reason about in
   better isolation.
   
   ### Jira
   
   - [x] My PR addresses 
[AVRO-2748](https://issues.apache.org/jira/browse/AVRO-2748).
   - [x] My PR references the ticket in its title.
   - [x] My PR adds no dependencies.
   
   ### Tests
   
   - [x] My PR adds no new tests; however the change is covered by tests.
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines.
   
   ### Documentation
   
   - [x] My PR adds no new functionality.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] cutting commented on a change in pull request #842: AVRO-2723: Refactor ReflectData to allow derived classes to customize default values for fields

[GitBox]

cutting commented on a change in pull request #842:
URL: https://github.com/apache/avro/pull/842#discussion_r413383635



##
File path: lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
##
@@ -500,6 +582,37 @@ static boolean isNonStringMapSchema(Schema s) {
 return false;
   }
 
+  /**
+   * Get default value for a schema field. Derived classes can override this
+   * method to provide values based on object instantiation
+   *
+   * @param typeType
+   * @param field   Field
+   * @param fieldSchema Schema of the field
+   * @return The default value
+   */
+  protected Object createSchemaDefaultValue(Type type, Field field, Schema 
fieldSchema) {
+Object defaultValue;
+if (defaultGenerated) {
+  defaultValue = getOrCreateDefaultValue(type, field);
+  if (defaultValue != null) {
+return defaultValue;

Review comment:
   Should we instead return deepCopy(fieldSchema, defaultValue)?

##
File path: lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
##
@@ -110,14 +112,96 @@ public static ReflectData get() {
   }
 
   /**
-   * Cause a class to be treated as though it had an {@link Stringable}
-   ** annotation.
+   * Cause a class to be treated as though it had an {@link Stringable} *
+   * annotation.
*/
   public ReflectData addStringable(Class c) {
 stringableClasses.add(c);
 return this;
   }
 
+  /**
+   * If this flag is set to true, default values for fields will be assigned
+   * dynamically using Java reflections. Let's call this feature `default
+   * reflection`. Initially this feature is disabled
+   */
+  private boolean defaultGenerated = false;
+
+  /**
+   * Enable or disable `default reflection`
+   *
+   * @param enabled set to `true` to enable the feature. This feature is 
disabled
+   *by default
+   * @return The current instance
+   */
+  public ReflectData setDefaultsGenerated(boolean enabled) {
+this.defaultGenerated = enabled;
+return this;
+  }
+
+  private final Map defaultValues = new WeakHashMap<>();

Review comment:
   This needs to be  so that defaults are GC'd when a class 
is.  Also, Class should probably be used instead of String for classes in other 
code below.

##
File path: lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
##
@@ -110,14 +112,96 @@ public static ReflectData get() {
   }
 
   /**
-   * Cause a class to be treated as though it had an {@link Stringable}
-   ** annotation.
+   * Cause a class to be treated as though it had an {@link Stringable} *
+   * annotation.
*/
   public ReflectData addStringable(Class c) {
 stringableClasses.add(c);
 return this;
   }
 
+  /**
+   * If this flag is set to true, default values for fields will be assigned
+   * dynamically using Java reflections. Let's call this feature `default

Review comment:
   We might say a bit more about how this works, that, when enabled, 
defaults are the field values of an instance created with a no-arg constructor.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AVRO-2765) dump command in avro-python3 tool module fails

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089990#comment-17089990
 ] 

Hudson commented on AVRO-2765:
--

SUCCESS: Integrated in Jenkins build AvroJava #845 (See 
[https://builds.apache.org/job/AvroJava/845/])
AVRO-2765: Py3 Support: Use open(), not file() (#836) (github: 
[https://github.com/apache/avro/commit/97c2aeb6ca5186524e8576febfe44b1a03a1b97e])
* (edit) lang/py/avro/tool.py
* (edit) lang/py3/avro/tool.py


> dump command in avro-python3 tool module fails
> --
>
> Key: AVRO-2765
> URL: https://issues.apache.org/jira/browse/AVRO-2765
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python, tools
>Affects Versions: 1.9.2
>Reporter: Olaf
>Assignee: Michael A. Smith
>Priority: Minor
> Fix For: 1.10.0
>
>
> Trying to dump an Avro file with the python3 tool command:
> {quote}{{> python3 -m *avro.tool* dump test.avro}}
> {quote}
> fails on:
> {quote}{{NameError: name 'file' is not defined}}
> {quote}
> This needs fixing by changing line 105 of avro/tool.py from:
> {quote}{{return *file*(f)}}
> {quote}
> into:
> {quote}{{return *open*(f, 'rb')}}
> {quote}
> This affects (most recent) Python3 {{_avro-python3_}} PyPI version 1.9.2.1. 
> Didn't check whether older Python3 versions and Python2 {{_avro_}} PyPI 
> version suffer from the same issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2801) Cache Hashcode of UTF8 Strings in all Set Methods

[Hudson (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089991#comment-17089991
 ] 

Hudson commented on AVRO-2801:
--

SUCCESS: Integrated in Jenkins build AvroJava #845 (See 
[https://builds.apache.org/job/AvroJava/845/])
AVRO-2801: Cache Hashcode of UTF8 Strings in all Set Methods (#856) (github: 
[https://github.com/apache/avro/commit/8e345c48e8f9708c27af63754965777314dac85c])
* (edit) lang/java/avro/src/main/java/org/apache/avro/util/Utf8.java


> Cache Hashcode of UTF8 Strings in all Set Methods
> -
>
> Key: AVRO-2801
> URL: https://issues.apache.org/jira/browse/AVRO-2801
> Project: Apache Avro
>  Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AVRO-2704) Cache Hashcode of UTF8 Strings

[Fokko Driesprong (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2704.

Resolution: Fixed

> Cache Hashcode of UTF8 Strings
> --
>
> Key: AVRO-2704
> URL: https://issues.apache.org/jira/browse/AVRO-2704
> Project: Apache Avro
>  Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
>
> Cache computed hash value for {{Utf8}} string instead of computing it every 
> time it is required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2801) Cache Hashcode of UTF8 Strings in all Set Methods

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089970#comment-17089970
 ] 

ASF subversion and git services commented on AVRO-2801:
---

Commit 8e345c48e8f9708c27af63754965777314dac85c in avro's branch 
refs/heads/master from belugabehr
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=8e345c4 ]

AVRO-2801: Cache Hashcode of UTF8 Strings in all Set Methods (#856)

Co-authored-by: David Mollitor 

> Cache Hashcode of UTF8 Strings in all Set Methods
> -
>
> Key: AVRO-2801
> URL: https://issues.apache.org/jira/browse/AVRO-2801
> Project: Apache Avro
>  Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (AVRO-2801) Cache Hashcode of UTF8 Strings in all Set Methods

[Fokko Driesprong (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2801.

Fix Version/s: 1.10.0
   Resolution: Fixed

> Cache Hashcode of UTF8 Strings in all Set Methods
> -
>
> Key: AVRO-2801
> URL: https://issues.apache.org/jira/browse/AVRO-2801
> Project: Apache Avro
>  Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [avro] Fokko commented on issue #858: AVRO-2790:Necessity of using Object as parameter of javaEscape method

[GitBox]

Fokko commented on issue #858:
URL: https://github.com/apache/avro/pull/858#issuecomment-617981040


   I like the change, more specific is better :-) Can you rebase onto the 
latest master? The CI seems to fail for an unobvious reason.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] Fokko commented on issue #855: AVRO-2800 validate enum symbol values

[GitBox]

Fokko commented on issue #855:
URL: https://github.com/apache/avro/pull/855#issuecomment-617972136


   @tjwp Do you have time to shed some light on this? :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] Fokko commented on issue #680: AVRO-2600: Swap to new properties in Velocity Engine

[GitBox]

Fokko commented on issue #680:
URL: https://github.com/apache/avro/pull/680#issuecomment-617971400


   @maisreymom Can you share the errors?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (AVRO-2765) dump command in avro-python3 tool module fails

[Fokko Driesprong (Jira)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2765.

Fix Version/s: 1.10.0
 Assignee: Michael A. Smith
   Resolution: Fixed

> dump command in avro-python3 tool module fails
> --
>
> Key: AVRO-2765
> URL: https://issues.apache.org/jira/browse/AVRO-2765
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python, tools
>Affects Versions: 1.9.2
>Reporter: Olaf
>Assignee: Michael A. Smith
>Priority: Minor
> Fix For: 1.10.0
>
>
> Trying to dump an Avro file with the python3 tool command:
> {quote}{{> python3 -m *avro.tool* dump test.avro}}
> {quote}
> fails on:
> {quote}{{NameError: name 'file' is not defined}}
> {quote}
> This needs fixing by changing line 105 of avro/tool.py from:
> {quote}{{return *file*(f)}}
> {quote}
> into:
> {quote}{{return *open*(f, 'rb')}}
> {quote}
> This affects (most recent) Python3 {{_avro-python3_}} PyPI version 1.9.2.1. 
> Didn't check whether older Python3 versions and Python2 {{_avro_}} PyPI 
> version suffer from the same issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2765) dump command in avro-python3 tool module fails

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089943#comment-17089943
 ] 

ASF subversion and git services commented on AVRO-2765:
---

Commit 97c2aeb6ca5186524e8576febfe44b1a03a1b97e in avro's branch 
refs/heads/master from Michael A. Smith
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=97c2aeb ]

AVRO-2765: Py3 Support: Use open(), not file() (#836)



> dump command in avro-python3 tool module fails
> --
>
> Key: AVRO-2765
> URL: https://issues.apache.org/jira/browse/AVRO-2765
> Project: Apache Avro
>  Issue Type: Bug
>  Components: python, tools
>Affects Versions: 1.9.2
>Reporter: Olaf
>Priority: Minor
>
> Trying to dump an Avro file with the python3 tool command:
> {quote}{{> python3 -m *avro.tool* dump test.avro}}
> {quote}
> fails on:
> {quote}{{NameError: name 'file' is not defined}}
> {quote}
> This needs fixing by changing line 105 of avro/tool.py from:
> {quote}{{return *file*(f)}}
> {quote}
> into:
> {quote}{{return *open*(f, 'rb')}}
> {quote}
> This affects (most recent) Python3 {{_avro-python3_}} PyPI version 1.9.2.1. 
> Didn't check whether older Python3 versions and Python2 {{_avro_}} PyPI 
> version suffer from the same issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [avro] Fokko commented on issue #836: AVRO-2765: Py3 Support: Use open(), not file()

[GitBox]

Fokko commented on issue #836:
URL: https://github.com/apache/avro/pull/836#issuecomment-617964021


   Thanks again @kojiromike 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] PaluruSumanth edited a comment on issue #521: AVRO-2389: CSharp serializers for plain C# objects

[GitBox]

PaluruSumanth edited a comment on issue #521:
URL: https://github.com/apache/avro/pull/521#issuecomment-617806930


   @blachniet  Though we have the support for .net framework 4.6.1, it also has 
the dependency on net standard framework, we see that target frameworks doesnt 
have .net framework 4.6.1, is that intended?
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] PaluruSumanth commented on issue #521: AVRO-2389: CSharp serializers for plain C# objects

[GitBox]

PaluruSumanth commented on issue #521:
URL: https://github.com/apache/avro/pull/521#issuecomment-617806930


   @blachniet  Though we have the support for .net framework 4.6.1, it also has 
the dependency on net standard library, we see that target frameworks doesnt 
have .net framework 4.6.1, is that intended?
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Proposal: RFCs for Avro 2.x

[Andy Le]

Hi guys,

I'm new to this vibrant open source community. My story with Avro can be found 
here [1]

While implementing the feature, I got stuck and had various discussions with 
Dough Cutting, Fokko Driesprong You may see here [2]

Here my (bias) observations about our current Avro 1.9.x:

- Some improvements can't be made due to fear of backward incompatibilities. 
For example: specifications about named Union.

- If `Apache Avro™ is a data serialization system.` then the repository 
`apache/avro` should solely focus on (de)serialization, right? Currently our 
repository contains many nice-to-have-but-not-critical things like: File I/O, 
Network I/O

IMHO, I think:

- We should publicly gather RFCs for Avro 2.x

- We should move such nice things out of Avro 2.x (may be to other dedicated 
repositories)

What do you think about my suggestions. Pls kindly let me know.

Thank you & be strong.

[1] My fork: https://github.com/anhldbk/avro-fork#why-this-fork
[2] My opened issue: 
https://issues.apache.org/jira/browse/AVRO-2808?jql=reporter%3Danhldbk%20AND%20resolution%20is%20EMPTY

[GitHub] [avro] anhldbk commented on issue #842: AVRO-2723: Refactor ReflectData to allow derived classes to customize default values for fields

[GitBox]

anhldbk commented on issue #842:
URL: https://github.com/apache/avro/pull/842#issuecomment-617786652


   @cutting Thank you for your thorough comments. I've fixed them. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [avro] kojiromike commented on issue #836: AVRO-2765: Py3 Support: Use open(), not file()

[GitBox]

kojiromike commented on issue #836:
URL: https://github.com/apache/avro/pull/836#issuecomment-617760289


   Huh, I haven't been able to reproduce that problem locally. Hopefully it was 
just a fluke.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (AVRO-248) make unions a named type

[Elliot West (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089535#comment-17089535
 ] 

Elliot West edited comment on AVRO-248 at 4/22/20, 11:15 AM:
-

I see this issue is quite old, but I am wondering if there would be any 
interest in adding this to the specification and implementing it? Specifically, 
I'm thinking about this kind of construct as described previously by [~cutting]:
{code:java}
{
  "type": "union",
  "name": "Foo",
  "branches": [
"string",
"Bar",
...
  ]
}{code}
The reason I ask is that I believe that there are new use-cases that could 
greatly benefit from this feature, specifically those that currently require 
[multi-typed streams in 
Kafka|https://www.confluent.io/blog/put-several-event-types-kafka-topic/] or 
indeed any streaming platform. There is already [an alternative 
implementation|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090]
 for this functionality, but this sits outside of Avro and in my opinion a 
sub-optimal work-around with [a number of significant 
issues|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090].

I would suggest that by implementing this feature in Avro, we can fully satisfy 
multi-typed stream use-cases in a clean, simple, and elegant manner, without 
needing to build out external implementations that attempt to work around this 
absent Avro feature.

 


was (Author: teabot):
I see this issue is quite old, but I am wondering if there would be any 
interest in adding this to the specification and implementing it? Specifically, 
I'm thinking about this kind of construct as described previously by [~cutting]:
{code:java}
{
  "type": "union",
  "name": "Foo",
  "branches": [
"string",
"Bar",
...
  ]
}{code}
The reason I ask is that I believe that there are new use-cases that could 
greatly benefit from this feature, specifically those that currently require 
[multi-typed streams in 
Kafka|https://www.confluent.io/blog/put-several-event-types-kafka-topic/] or 
indeed any streaming platform. There is already [an alternative implementation 
for 
this|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090]
 for this functionality, but this sits outside of Avro and in my opinion a 
sub-optimal work-around with [a number of significant 
issues|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090].

I would suggest that by implementing this feature in Avro, we can fully satisfy 
multi-typed stream use-cases in a clean, simple, and elegant manner, without 
needing to build out external implementations that attempt to work around this 
absent Avro feature.

 

> make unions a named type
> 
>
> Key: AVRO-248
> URL: https://issues.apache.org/jira/browse/AVRO-248
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: spec
>Reporter: Doug Cutting
>Priority: Major
>
> Unions are currently anonymous.  However it might be convenient if they were 
> named.  In particular:
>  - when code is generated for a union, a class could be generated that 
> includes an enum indicating which branch of the union is taken, e.g., a union 
> of string and int named Foo might cause a Java class like {code}
> public class Foo {
>   public static enum Type {STRING, INT};
>   private Type type;
>   private Object datum;
>   public Type getType();
>   public String getString() { if (type==STRING) return (String)datum; else 
> throw ... }
>   public void setString(String s) { type = STRING;  datum = s; }
>   
> }
> {code} Then Java applications can easily use a switch statement to process 
> union values rather than using instanceof.
>  - when using reflection, an abstract class with a set of concrete 
> implementations can be represented as a union (AVRO-241).  However, if one 
> wishes to create an array one must know the name of the base class, which is 
> not represented in the Avro schema.  One approach would be to add an 
> annotation to the reflected array schema (AVRO-242) noting the base class.  
> But if the union itself were named, that could name the base class.  This 
> would also make reflected protocol interfaces more consise, since the base 
> class name could be used in parameters return types and fields.
>  - Generalizing the above: Avro lacks class inheritance, unions are a way to 
> model inheritance, and this model is more useful if the union is named.
> This would be an incompatible change to schemas.  If we go this way, we 
> should probably rename 1.3 to 2.0.  Note that AVRO-160 proposes an 
> incompatible change to data file formats, which may also force a major 
> release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-248) make unions a named type

[Andy Le (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089555#comment-17089555
 ] 

Andy Le commented on AVRO-248:
--

[~teabot] I think it's a good time to have RFCs for Avro 2.x

> make unions a named type
> 
>
> Key: AVRO-248
> URL: https://issues.apache.org/jira/browse/AVRO-248
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: spec
>Reporter: Doug Cutting
>Priority: Major
>
> Unions are currently anonymous.  However it might be convenient if they were 
> named.  In particular:
>  - when code is generated for a union, a class could be generated that 
> includes an enum indicating which branch of the union is taken, e.g., a union 
> of string and int named Foo might cause a Java class like {code}
> public class Foo {
>   public static enum Type {STRING, INT};
>   private Type type;
>   private Object datum;
>   public Type getType();
>   public String getString() { if (type==STRING) return (String)datum; else 
> throw ... }
>   public void setString(String s) { type = STRING;  datum = s; }
>   
> }
> {code} Then Java applications can easily use a switch statement to process 
> union values rather than using instanceof.
>  - when using reflection, an abstract class with a set of concrete 
> implementations can be represented as a union (AVRO-241).  However, if one 
> wishes to create an array one must know the name of the base class, which is 
> not represented in the Avro schema.  One approach would be to add an 
> annotation to the reflected array schema (AVRO-242) noting the base class.  
> But if the union itself were named, that could name the base class.  This 
> would also make reflected protocol interfaces more consise, since the base 
> class name could be used in parameters return types and fields.
>  - Generalizing the above: Avro lacks class inheritance, unions are a way to 
> model inheritance, and this model is more useful if the union is named.
> This would be an incompatible change to schemas.  If we go this way, we 
> should probably rename 1.3 to 2.0.  Note that AVRO-160 proposes an 
> incompatible change to data file formats, which may also force a major 
> release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (AVRO-248) make unions a named type

[Elliot West (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089535#comment-17089535
 ] 

Elliot West edited comment on AVRO-248 at 4/22/20, 10:33 AM:
-

I see this issue is quite old, but I am wondering if there would be any 
interest in adding this to the specification and implementing it? Specifically, 
I'm thinking about this kind of construct as described previously by [~cutting]:
{code:java}
{
  "type": "union",
  "name": "Foo",
  "branches": [
"string",
"Bar",
...
  ]
}{code}
The reason I ask is that I believe that there are new use-cases that could 
greatly benefit from this feature, specifically those that currently require 
[multi-typed streams in 
Kafka|https://www.confluent.io/blog/put-several-event-types-kafka-topic/] or 
indeed any streaming platform. There is already [an alternative implementation 
for 
this|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090]
 for this functionality, but this sits outside of Avro and in my opinion a 
sub-optimal work-around with [a number of significant 
issues|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090].

I would suggest that by implementing this feature in Avro, we can fully satisfy 
multi-typed stream use-cases in a clean, simple, and elegant manner, without 
needing to build out external implementations that attempt to work around this 
absent Avro feature.

 


was (Author: teabot):
I see this issue is quite old, but I am wondering if there would be any 
interest in adding this to the specification and implementing it? Specifically, 
I'm thinking about this kind of construct as described previously by [~cutting]:

 
{code:java}
{
  "type": "union",
  "name": "Foo",
  "branches": [
"string",
"Bar",
...
  ]
}{code}
 

The reason I ask is that I believe that there are new use-cases that could 
greatly benefit from this feature, specifically those that currently require 
[multi-typed streams in 
Kafka|https://www.confluent.io/blog/put-several-event-types-kafka-topic/] or 
indeed any streaming platform. There is already [an alternative implementation 
for 
this|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090]
 for this functionality, but this sits outside of Avro and in my opinion a 
sub-optimal work-around with [a number of significant 
issues|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090].

I would suggest that by implementing this feature in Avro, we can fully satisfy 
multi-typed stream use-cases in a clean, simple, and elegant manner, without 
needing to build out external implementations that attempt to work around this 
absent Avro feature.

 

> make unions a named type
> 
>
> Key: AVRO-248
> URL: https://issues.apache.org/jira/browse/AVRO-248
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: spec
>Reporter: Doug Cutting
>Priority: Major
>
> Unions are currently anonymous.  However it might be convenient if they were 
> named.  In particular:
>  - when code is generated for a union, a class could be generated that 
> includes an enum indicating which branch of the union is taken, e.g., a union 
> of string and int named Foo might cause a Java class like {code}
> public class Foo {
>   public static enum Type {STRING, INT};
>   private Type type;
>   private Object datum;
>   public Type getType();
>   public String getString() { if (type==STRING) return (String)datum; else 
> throw ... }
>   public void setString(String s) { type = STRING;  datum = s; }
>   
> }
> {code} Then Java applications can easily use a switch statement to process 
> union values rather than using instanceof.
>  - when using reflection, an abstract class with a set of concrete 
> implementations can be represented as a union (AVRO-241).  However, if one 
> wishes to create an array one must know the name of the base class, which is 
> not represented in the Avro schema.  One approach would be to add an 
> annotation to the reflected array schema (AVRO-242) noting the base class.  
> But if the union itself were named, that could name the base class.  This 
> would also make reflected protocol interfaces more consise, since the base 
> class name could be used in parameters return types and fields.
>  - Generalizing the above: Avro lacks class inheritance, unions are a way to 
> model inheritance, and this model is more useful if the union is named.
> This would be an incompatible change to schemas.  If we go this way, we 
> should probably rename 1.3 to 2.0.  Note that AVRO-160 proposes an 
> incompatible change to data file formats, which may also force a major 
> release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-248) make unions a named type

[Elliot West (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089535#comment-17089535
 ] 

Elliot West commented on AVRO-248:
--

I see this issue is quite old, but I am wondering if there would be any 
interest in adding this to the specification and implementing it? Specifically, 
I'm thinking about this kind of construct as described previously by [~cutting]:

 
{code:java}
{
  "type": "union",
  "name": "Foo",
  "branches": [
"string",
"Bar",
...
  ]
}{code}
 

The reason I ask is that I believe that there are new use-cases that could 
greatly benefit from this feature, specifically those that currently require 
[multi-typed streams in 
Kafka|https://www.confluent.io/blog/put-several-event-types-kafka-topic/] or 
indeed any streaming platform. There is already [an alternative implementation 
for 
this|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090]
 for this functionality, but this sits outside of Avro and in my opinion a 
sub-optimal work-around with [a number of significant 
issues|https://github.com/confluentinc/schema-registry/pull/680#issuecomment-511796090].

I would suggest that by implementing this feature in Avro, we can fully satisfy 
multi-typed stream use-cases in a clean, simple, and elegant manner, without 
needing to build out external implementations that attempt to work around this 
absent Avro feature.

 

> make unions a named type
> 
>
> Key: AVRO-248
> URL: https://issues.apache.org/jira/browse/AVRO-248
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: spec
>Reporter: Doug Cutting
>Priority: Major
>
> Unions are currently anonymous.  However it might be convenient if they were 
> named.  In particular:
>  - when code is generated for a union, a class could be generated that 
> includes an enum indicating which branch of the union is taken, e.g., a union 
> of string and int named Foo might cause a Java class like {code}
> public class Foo {
>   public static enum Type {STRING, INT};
>   private Type type;
>   private Object datum;
>   public Type getType();
>   public String getString() { if (type==STRING) return (String)datum; else 
> throw ... }
>   public void setString(String s) { type = STRING;  datum = s; }
>   
> }
> {code} Then Java applications can easily use a switch statement to process 
> union values rather than using instanceof.
>  - when using reflection, an abstract class with a set of concrete 
> implementations can be represented as a union (AVRO-241).  However, if one 
> wishes to create an array one must know the name of the base class, which is 
> not represented in the Avro schema.  One approach would be to add an 
> annotation to the reflected array schema (AVRO-242) noting the base class.  
> But if the union itself were named, that could name the base class.  This 
> would also make reflected protocol interfaces more consise, since the base 
> class name could be used in parameters return types and fields.
>  - Generalizing the above: Avro lacks class inheritance, unions are a way to 
> model inheritance, and this model is more useful if the union is named.
> This would be an incompatible change to schemas.  If we go this way, we 
> should probably rename 1.3 to 2.0.  Note that AVRO-160 proposes an 
> incompatible change to data file formats, which may also force a major 
> release.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-2806) Upgrade Netty to 4.x

[Gabor Szadovszky (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089468#comment-17089468
 ] 

Gabor Szadovszky commented on AVRO-2806:


[~zeshuai007], as you've said there are many major changes between 3.x and 4.x. 
I don't really remember what were the exact issues but I could not refactor our 
code to use the 4.x. Please note, that I have no experience with Netty. My only 
motivation are the CVEs reported for 3.x.
Seems that {{netty-codec-http2}} was introduced separately from {{netty}} in 
AVRO-2279. I don't know why don't we use the same version or if it matters at 
all.

> Upgrade Netty to 4.x
> 
>
> Key: AVRO-2806
> URL: https://issues.apache.org/jira/browse/AVRO-2806
> Project: Apache Avro
>  Issue Type: Task
>  Components: java
>Affects Versions: 1.9.2
>Reporter: Gabor Szadovszky
>Priority: Major
>
> ipc-netty still uses the 3.x line which is ended almost 4 years ago. We have 
> a couple of CVEs which won't be fixed in 3.x so it is important to migrate to 
> the 4.x.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (AVRO-2811) Upgrade grpc 1.26 to 1.28

[Zezeng Wang (Jira)]

Zezeng Wang created AVRO-2811:
-

 Summary: Upgrade grpc 1.26 to 1.28
 Key: AVRO-2811
 URL: https://issues.apache.org/jira/browse/AVRO-2811
 Project: Apache Avro
  Issue Type: Task
Affects Versions: 1.9.2
Reporter: Zezeng Wang
 Fix For: 1.9.3


In netty 1.26 there are bugs that need attention.
Eg: [Deadlock on start gRPC 
server|https://github.com/grpc/grpc-java/issues/6601]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)