[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804811 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 30/Aug/22 12:12 Start Date: 30/Aug/22 12:12 Worklog Time Spent: 10m Work Description: martin-g commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1231580851 Thank you for the contribution, @rayokota ! Thanks to all reviews too! Issue Time Tracking --- Worklog Id: (was: 804811) Time Spent: 11h (was: 10h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > Time Spent: 11h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804810&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804810 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 30/Aug/22 12:10 Start Date: 30/Aug/22 12:10 Worklog Time Spent: 10m Work Description: martin-g commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r958398630 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: Done with a60b748c Issue Time Tracking --- Worklog Id: (was: 804810) Time Spent: 10h 50m (was: 10h 40m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804803 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 30/Aug/22 12:04 Start Date: 30/Aug/22 12:04 Worklog Time Spent: 10m Work Description: martin-g merged PR #1833: URL: https://github.com/apache/avro/pull/1833 Issue Time Tracking --- Worklog Id: (was: 804803) Time Spent: 10h 40m (was: 10.5h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804772 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 30/Aug/22 10:13 Start Date: 30/Aug/22 10:13 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r958289034 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: That seems right Issue Time Tracking --- Worklog Id: (was: 804772) Time Spent: 10.5h (was: 10h 20m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10.5h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804766&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804766 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 30/Aug/22 09:57 Start Date: 30/Aug/22 09:57 Worklog Time Spent: 10m Work Description: martin-g commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r958272425 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: So, this PR could be merged as is only in 1.12.0. For branch-1.11 we should cherry-pick the squashed commit from `master`, remove this method and add an extra commit, right ? Issue Time Tracking --- Worklog Id: (was: 804766) Time Spent: 10h 20m (was: 10h 10m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10h 20m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804723
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 30/Aug/22 08:50
Start Date: 30/Aug/22 08:50
Worklog Time Spent: 10m
Work Description: martin-g commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r958196081
##
lang/csharp/src/apache/test/IO/JsonCodecTests.cs:
##
@@ -0,0 +1,329 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using NUnit.Framework;
+using System.IO;
+using System.Linq;
+using System.Text;
+using Avro.Generic;
+using Avro.IO;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
+
+namespace Avro.Test
+{
+using Decoder = Avro.IO.Decoder;
+using Encoder = Avro.IO.Encoder;
+
+///
+/// Tests the JsonEncoder and JsonDecoder.
+///
+[TestFixture]
+public class JsonCodecTests
+{
+[TestCase("{ \"type\": \"record\", \"name\": \"r\", \"fields\": [ " +
+ " { \"name\" : \"f1\", \"type\": \"int\" }, " +
+ " { \"name\" : \"f2\", \"type\": \"float\" } " +
+ "] }",
+"{ \"f2\": 10.4, \"f1\": 10 } ")]
+[TestCase("{ \"type\": \"enum\", \"name\": \"e\", \"symbols\": " + "[
\"s1\", \"s2\"] }", " \"s1\" ")]
+[TestCase("{ \"type\": \"enum\", \"name\": \"e\", \"symbols\": " + "[
\"s1\", \"s2\"] }", " \"s2\" ")]
+[TestCase("{ \"type\": \"fixed\", \"name\": \"f\", \"size\": 5 }",
"\"hello\"")]
+[TestCase("{ \"type\": \"array\", \"items\": \"int\" }", "[ 10, 20, 30
]")]
+[TestCase("{ \"type\": \"map\", \"values\": \"int\" }", "{ \"k1\": 10,
\"k2\": 20, \"k3\": 30 }")]
+[TestCase("[ \"int\", \"long\" ]", "{ \"int\": 10 }")]
+[TestCase("\"string\"", "\"hello\"")]
+[TestCase("\"bytes\"", "\"hello\"")]
+[TestCase("\"int\"", "10")]
+[TestCase("\"long\"", "10")]
+[TestCase("\"float\"", "10.0")]
+[TestCase("\"double\"", "10.0")]
+[TestCase("\"boolean\"", "true")]
+[TestCase("\"boolean\"", "false")]
+[TestCase("\"null\"", "null")]
+public void TestJsonAllTypesValidValues(String schemaStr, String value)
+{
+Schema schema = Schema.Parse(schemaStr);
+byte[] avroBytes = fromJsonToAvro(value, schema);
+
+Assert.IsTrue(JToken.DeepEquals(JToken.Parse(value),
+JToken.Parse(fromAvroToJson(avroBytes, schema, true;
+}
+
+[TestCase("{ \"type\": \"record\", \"name\": \"r\", \"fields\": [ " +
+ " { \"name\" : \"f1\", \"type\": \"int\" }, " +
+ " { \"name\" : \"f2\", \"type\": \"float\" } " +
+ "] }",
+"{ \"f4\": 10.4, \"f3\": 10 } ")]
+[TestCase("{ \"type\": \"enum\", \"name\": \"e\", \"symbols\": " + "[
\"s1\", \"s2\"] }", " \"s3\" ")]
Review Comment:
```suggestion
[TestCase("{ \"type\": \"enum\", \"name\": \"e\", \"symbols\": [
\"s1\", \"s2\"] }", " \"s3\" ")]
```
##
lang/csharp/src/apache/test/IO/JsonCodecTests.cs:
##
@@ -0,0 +1,329 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using NUnit.Framework;
+using System
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804516 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 29/Aug/22 19:15 Start Date: 29/Aug/22 19:15 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1230744178 Thanks @martin-g . I added unit tests to test all Avro types, both with valid and invalid values, as well as tests for malformed JSON. Issue Time Tracking --- Worklog Id: (was: 804516) Time Spent: 10h (was: 9h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 10h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804274 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 29/Aug/22 07:28 Start Date: 29/Aug/22 07:28 Worklog Time Spent: 10m Work Description: martin-g commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1229887817 > My biggest concern is that user are not able inject own custom logic without changing library. If you add more interfaces and use interface instead of concert classes as dependency, it allows users who want benefit from dependency injection use it. As far as I understand the code the encoder/decoder code does not use `static` methods/fields and it is easily extendable/overrideable. `Symbol` class has `static`s indeed! Although they could be explained as an implementation detail of the JsonDecoder I agree that it would be better if their usage is reduced as much as possible. For example adding unit tests for each new class (like JsonDecoderTests.cs, JsonEncoderTests.cs, Symbol.cs) would show us what is well designed and extendable, and what is problematic due to the usage of `static`s. At the moment we have only JsonCodecTests.cs which is rather higher level / functional test. To summarize my review: I think the PR needs more unit tests, especially ones testing invalid input: wrong JSON, wrong values for the expected schema, etc. Issue Time Tracking --- Worklog Id: (was: 804274) Time Spent: 9h 50m (was: 9h 40m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 9h 50m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804118 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 27/Aug/22 06:26 Start Date: 27/Aug/22 06:26 Worklog Time Spent: 10m Work Description: KhrystynaPopadyuk commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1229134158 Hi @rayokota , All comments below, are suggestion only. My biggest concern is that user are not able inject own custom logic without changing libraries. If you add more interfaces and use interface instead of concert classes as dependency, it allows users who want benefit from dependency injection use it. Issue Time Tracking --- Worklog Id: (was: 804118) Time Spent: 9h 40m (was: 9.5h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 9h 40m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804117&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804117 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 27/Aug/22 06:07 Start Date: 27/Aug/22 06:07 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1229131964 > Hi @martin-g and @rayokota , Hi @KhrystynaPopadyuk , > > Please find my 5 cents below. > > As was mentioned in description this is port from Java and it looks and feels exactly as port. Of course this code will work. But it would be very difficult maintain, extend and provide sufficient unit test coverage. This is new functionality and it is easy refactor it now, and would be very difficult update if later. > > Please find below few suggestion that, I believe, should be addressed: > > 1. Folder and file structure. For example: > > * Why do not create separate folder for JsonParser and placed all related classes there All the parsing classes are indeed in a separate `Parsing` package. The `JsonEncoder` and `JsonDecoder` are siblings of `BinaryEncoder` and `BinaryDecoder` in the `IO` package, which makes sense to me. > * "ParsingDecoder" (https://github.com/apache/avro/blob/c13a45debf4594dc7bc3c90aef64cee5ae115a5d/lang/csharp/src/apache/main/IO/ParsingDecoder.cs) It states that this is "Base class for a -based" but Parser (https://github.com/apache/avro/blob/c13a45debf4594dc7bc3c90aef64cee5ae115a5d/lang/csharp/src/apache/main/IO/Parsing/Parser.cs) does no have base class. Moreover ParsingDecoder is base class for JsonDecoder. So is "ParsingDecoder " correct name? What is the purpose of having this class? Why we need this additional layer of abstraction? The `ParsingEncoder` and `ParsingDecoder` are implementations of `Encoder` and `Decoder` that maintain some parsing state. The `JsonEncoder` and `JsonDecoder` subclass those classes to delegate the maintenance of the parsing state. In the Java code, there are also `ValidatingEncoder` and `ValidatingDecoder` that subclass `ParsingEncoder` and `ParsingDecoder`. I did not port those classes over. > * there are a lot of child classes and interfaces. For example file Symbol.cs contains not only abstract class Symbol but 17 child classes and one enum, And all definitions of fields, properties, child classes, enums, methods are without any structure and order. The `Symbol` class maintains the terminals and non-terminals of the parsing grammar, so that is why there are a lot of classes and enums in it. > * there are a lot of other things that need attention > > 2. Static method and classes is very controversial tool. It has own benefits but also it can brink a lot of negative. Overall we should be use them carefully and with attention. I have founded 38 static items in Symbol class. That looks very suspicious. I agree static methods and classes have their advantages and disadvantages, but I believe there is nothing wrong in the use of them in the `Symbol` class. I'd be happy to incorporate more specific suggestions at particular line numbers. > 3. Poor usage of interfaces but a lot of static and even hidden dependencies. That should be reviewed in direction to use interfaces as dependency instead of classes (even abstract). Dependency inversion/ dependency injection can bring a lot of benefit and flexibility to code. Dependency inversion also has its advantages and disadvantages, but I don't think it would really help here, unless you can be more specific. > 4. switch operator - is there way to reduce usage of switch/case? I think `switch` can also have its advantages and disadvantages, but I think its use is fine here. Issue Time Tracking --- Worklog Id: (was: 804117) Time Spent: 9.5h (was: 9h 20m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basic
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804104 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 27/Aug/22 02:28 Start Date: 27/Aug/22 02:28 Worklog Time Spent: 10m Work Description: KhrystynaPopadyuk commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1229103595 Hi @martin-g , Please find my 5 cents below. As was mentioned in description this is port from Java and it looks exactly as port. Of course this code will work. But it would be very difficult maintain, extend and provide sufficient unit test coverage. This is new functionality and it is easy refactor it now, and would be very difficult update if later. Please find below few suggestion and I believe should be address: 1. Folder and file structure. For example: - Why do not create separate folder for JsonParser and placed all related classes there - "ParsingDecoder" (https://github.com/apache/avro/blob/c13a45debf4594dc7bc3c90aef64cee5ae115a5d/lang/csharp/src/apache/main/IO/ParsingDecoder.cs) It stated that this is "Base class for a -based" but Parser (https://github.com/apache/avro/blob/c13a45debf4594dc7bc3c90aef64cee5ae115a5d/lang/csharp/src/apache/main/IO/Parsing/Parser.cs) does no have base class. Moreover ParsingDecoder is base class for JsonDecoder. So is "ParsingDecoder " correct name? What is the purpose of having this class? Why we need this additional layer of abstraction? - there are a lot of child classes and interfaces. For example file Symbol.cs contains not only abstract class Symbol but 17 child classes and one enum, And all definitions of fields, properties, child classes, enums, methods are without any structure and order. - there are a lot of other things that need attention 2. Static method and classes is very controversial tool. It has own benefits but also it can brink a lot of negative. Overall that should be use them carefully and with attention. I have founded 38 static items in Symbol class. That looks very suspicious. 3. Poor usage of interfaces but a lot of static and even hidden dependencies. That should be reviewed in direction to use interfaces as dependency instead of classes (even abstract). Dependency inversion/ dependency injection can bring a lot of benefit and flexibility to code. 4. switch operator - is there way to reduce usage of switch/case? Issue Time Tracking --- Worklog Id: (was: 804104) Time Spent: 9h 20m (was: 9h 10m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 9h 20m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804030&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804030
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 26/Aug/22 17:49
Start Date: 26/Aug/22 17:49
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r956273784
##
lang/csharp/src/apache/main/IO/Parsing/Symbol.cs:
##
@@ -0,0 +1,983 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Symbol is the base of all symbols (terminals and non-terminals) of the
+/// grammar.
+///
+public abstract class Symbol
+{
+///
+/// The type of symbol.
+///
+public enum Kind
+{
+///
+/// terminal symbols which have no productions
+Terminal,
+
+///
+/// Start symbol for some grammar
+Root,
+
+///
+/// non-terminal symbol which is a sequence of one or more other
symbols
+Sequence,
+
+///
+/// non-terminal to represent the contents of an array or map
+Repeater,
+
+///
+/// non-terminal to represent the union
+Alternative,
+
+///
+/// non-terminal action symbol which are automatically consumed
+ImplicitAction,
+
+///
+/// non-terminal action symbol which is explicitly consumed
+ExplicitAction
+}
+
+/// The kind of this symbol.
+public Kind SymKind { get; private set; }
+
+///
+/// The production for this symbol. If this symbol is a terminal this
is
+/// null. Otherwise this holds the the sequence of the
symbols that
+/// forms the production for this symbol. The sequence is in the
reverse order of
+/// production. This is useful for easy copying onto parsing stack.
+///
+/// Please note that this is a final. So the production for a symbol
should be
+/// known before that symbol is constructed. This requirement cannot
be met for
+/// those symbols which are recursive (e.g. a record that holds union
a branch of
+/// which is the record itself). To resolve this problem, we
initialize the
+/// symbol with an array of nulls. Later we fill the symbols. Not
clean, but
+/// works. The other option is to not have this field a final. But
keeping it
+/// final and thus keeping symbol immutable gives some comfort. See
various
+/// generators how we generate records.
+///
+public Symbol[] Production { get; private set; }
+
+///
+/// Constructs a new symbol of the given kind.
+///
+protected Symbol(Kind kind) : this(kind, null)
+{
+}
+
+///
+/// Constructs a new symbol of the given kind and production.
+///
+protected Symbol(Kind kind, Symbol[] production)
+{
+Production = production;
+SymKind = kind;
+}
+
+///
+/// A convenience method to construct a root symbol.
+///
+public static Symbol NewRoot(params Symbol[] symbols) => new
Root(symbols);
+
+///
+/// A convenience method to construct a sequence.
+///
+/// The constituent symbols of the sequence.
+public static Symbol NewSeq(params Symbol[] production) => new
Sequence(production);
+
+///
+/// A convenience method to construct a repeater.
+///
+/// The end symbol.
+/// The symbols to repeat in the repeater.
+public static Symbol NewRepeat(Symbol endSymbol, params Symbol[]
symsToRepeat) =>
+new Repeater(endSymbol, symsToRepeat);
+
+///
+
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804029&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804029 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 26/Aug/22 17:48 Start Date: 26/Aug/22 17:48 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1228762719 Thanks for the thorough review @KyleSchoonover , I've incorporated your latest feedback. Issue Time Tracking --- Worklog Id: (was: 804029) Time Spent: 9h (was: 8h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 9h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804028&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804028
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 26/Aug/22 17:47
Start Date: 26/Aug/22 17:47
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r956273784
##
lang/csharp/src/apache/main/IO/Parsing/Symbol.cs:
##
@@ -0,0 +1,983 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Symbol is the base of all symbols (terminals and non-terminals) of the
+/// grammar.
+///
+public abstract class Symbol
+{
+///
+/// The type of symbol.
+///
+public enum Kind
+{
+///
+/// terminal symbols which have no productions
+Terminal,
+
+///
+/// Start symbol for some grammar
+Root,
+
+///
+/// non-terminal symbol which is a sequence of one or more other
symbols
+Sequence,
+
+///
+/// non-terminal to represent the contents of an array or map
+Repeater,
+
+///
+/// non-terminal to represent the union
+Alternative,
+
+///
+/// non-terminal action symbol which are automatically consumed
+ImplicitAction,
+
+///
+/// non-terminal action symbol which is explicitly consumed
+ExplicitAction
+}
+
+/// The kind of this symbol.
+public Kind SymKind { get; private set; }
+
+///
+/// The production for this symbol. If this symbol is a terminal this
is
+/// null. Otherwise this holds the the sequence of the
symbols that
+/// forms the production for this symbol. The sequence is in the
reverse order of
+/// production. This is useful for easy copying onto parsing stack.
+///
+/// Please note that this is a final. So the production for a symbol
should be
+/// known before that symbol is constructed. This requirement cannot
be met for
+/// those symbols which are recursive (e.g. a record that holds union
a branch of
+/// which is the record itself). To resolve this problem, we
initialize the
+/// symbol with an array of nulls. Later we fill the symbols. Not
clean, but
+/// works. The other option is to not have this field a final. But
keeping it
+/// final and thus keeping symbol immutable gives some comfort. See
various
+/// generators how we generate records.
+///
+public Symbol[] Production { get; private set; }
+
+///
+/// Constructs a new symbol of the given kind.
+///
+protected Symbol(Kind kind) : this(kind, null)
+{
+}
+
+///
+/// Constructs a new symbol of the given kind and production.
+///
+protected Symbol(Kind kind, Symbol[] production)
+{
+Production = production;
+SymKind = kind;
+}
+
+///
+/// A convenience method to construct a root symbol.
+///
+public static Symbol NewRoot(params Symbol[] symbols) => new
Root(symbols);
+
+///
+/// A convenience method to construct a sequence.
+///
+/// The constituent symbols of the sequence.
+public static Symbol NewSeq(params Symbol[] production) => new
Sequence(production);
+
+///
+/// A convenience method to construct a repeater.
+///
+/// The end symbol.
+/// The symbols to repeat in the repeater.
+public static Symbol NewRepeat(Symbol endSymbol, params Symbol[]
symsToRepeat) =>
+new Repeater(endSymbol, symsToRepeat);
+
+///
+
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804024&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804024
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 26/Aug/22 17:25
Start Date: 26/Aug/22 17:25
Worklog Time Spent: 10m
Work Description: KalleOlaviNiemitalo commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r956252918
##
lang/csharp/src/apache/main/IO/Parsing/Symbol.cs:
##
@@ -0,0 +1,983 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Symbol is the base of all symbols (terminals and non-terminals) of the
+/// grammar.
+///
+public abstract class Symbol
+{
+///
+/// The type of symbol.
+///
+public enum Kind
+{
+///
+/// terminal symbols which have no productions
+Terminal,
+
+///
+/// Start symbol for some grammar
+Root,
+
+///
+/// non-terminal symbol which is a sequence of one or more other
symbols
+Sequence,
+
+///
+/// non-terminal to represent the contents of an array or map
+Repeater,
+
+///
+/// non-terminal to represent the union
+Alternative,
+
+///
+/// non-terminal action symbol which are automatically consumed
+ImplicitAction,
+
+///
+/// non-terminal action symbol which is explicitly consumed
+ExplicitAction
+}
+
+/// The kind of this symbol.
+public Kind SymKind { get; private set; }
+
+///
+/// The production for this symbol. If this symbol is a terminal this
is
+/// null. Otherwise this holds the the sequence of the
symbols that
+/// forms the production for this symbol. The sequence is in the
reverse order of
+/// production. This is useful for easy copying onto parsing stack.
+///
+/// Please note that this is a final. So the production for a symbol
should be
+/// known before that symbol is constructed. This requirement cannot
be met for
+/// those symbols which are recursive (e.g. a record that holds union
a branch of
+/// which is the record itself). To resolve this problem, we
initialize the
+/// symbol with an array of nulls. Later we fill the symbols. Not
clean, but
+/// works. The other option is to not have this field a final. But
keeping it
+/// final and thus keeping symbol immutable gives some comfort. See
various
+/// generators how we generate records.
+///
+public Symbol[] Production { get; private set; }
+
+///
+/// Constructs a new symbol of the given kind.
+///
+protected Symbol(Kind kind) : this(kind, null)
+{
+}
+
+///
+/// Constructs a new symbol of the given kind and production.
+///
+protected Symbol(Kind kind, Symbol[] production)
+{
+Production = production;
+SymKind = kind;
+}
+
+///
+/// A convenience method to construct a root symbol.
+///
+public static Symbol NewRoot(params Symbol[] symbols) => new
Root(symbols);
+
+///
+/// A convenience method to construct a sequence.
+///
+/// The constituent symbols of the sequence.
+public static Symbol NewSeq(params Symbol[] production) => new
Sequence(production);
+
+///
+/// A convenience method to construct a repeater.
+///
+/// The end symbol.
+/// The symbols to repeat in the repeater.
+public static Symbol NewRepeat(Symbol endSymbol, params Symbol[]
symsToRepeat) =>
+new Repeater(endSymbol, symsToRepeat);
+
+
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=804015&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804015
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 26/Aug/22 17:05
Start Date: 26/Aug/22 17:05
Worklog Time Spent: 10m
Work Description: KyleSchoonover commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r956225566
##
lang/csharp/src/apache/main/IO/Parsing/Symbol.cs:
##
@@ -0,0 +1,983 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Symbol is the base of all symbols (terminals and non-terminals) of the
+/// grammar.
+///
+public abstract class Symbol
+{
+///
+/// The type of symbol.
+///
+public enum Kind
+{
+///
+/// terminal symbols which have no productions
+Terminal,
+
+///
+/// Start symbol for some grammar
+Root,
+
+///
+/// non-terminal symbol which is a sequence of one or more other
symbols
+Sequence,
+
+///
+/// non-terminal to represent the contents of an array or map
+Repeater,
+
+///
+/// non-terminal to represent the union
+Alternative,
+
+///
+/// non-terminal action symbol which are automatically consumed
+ImplicitAction,
+
+///
+/// non-terminal action symbol which is explicitly consumed
+ExplicitAction
+}
+
+/// The kind of this symbol.
+public Kind SymKind { get; private set; }
+
+///
+/// The production for this symbol. If this symbol is a terminal this
is
+/// null. Otherwise this holds the the sequence of the
symbols that
+/// forms the production for this symbol. The sequence is in the
reverse order of
+/// production. This is useful for easy copying onto parsing stack.
+///
+/// Please note that this is a final. So the production for a symbol
should be
+/// known before that symbol is constructed. This requirement cannot
be met for
+/// those symbols which are recursive (e.g. a record that holds union
a branch of
+/// which is the record itself). To resolve this problem, we
initialize the
+/// symbol with an array of nulls. Later we fill the symbols. Not
clean, but
+/// works. The other option is to not have this field a final. But
keeping it
+/// final and thus keeping symbol immutable gives some comfort. See
various
+/// generators how we generate records.
+///
+public Symbol[] Production { get; private set; }
+
+///
+/// Constructs a new symbol of the given kind.
+///
+protected Symbol(Kind kind) : this(kind, null)
+{
+}
+
+///
+/// Constructs a new symbol of the given kind and production.
+///
+protected Symbol(Kind kind, Symbol[] production)
+{
+Production = production;
+SymKind = kind;
+}
+
+///
+/// A convenience method to construct a root symbol.
+///
+public static Symbol NewRoot(params Symbol[] symbols) => new
Root(symbols);
+
+///
+/// A convenience method to construct a sequence.
+///
+/// The constituent symbols of the sequence.
+public static Symbol NewSeq(params Symbol[] production) => new
Sequence(production);
+
+///
+/// A convenience method to construct a repeater.
+///
+/// The end symbol.
+/// The symbols to repeat in the repeater.
+public static Symbol NewRepeat(Symbol endSymbol, params Symbol[]
symsToRepeat) =>
+new Repeater(endSymbol, symsToRepeat);
+
+//
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803938&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803938 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 26/Aug/22 12:01 Start Date: 26/Aug/22 12:01 Worklog Time Spent: 10m Work Description: martin-g commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1228404830 @KhrystynaPopadyuk do want to join the reviewing process here ? It would be really nice if C# contributors help each other for the common good! @rayokota You could return the favour by reviewing others' PRs - https://github.com/apache/avro/pulls?q=is%3Aopen+is%3Apr+label%3AC%23 Issue Time Tracking --- Worklog Id: (was: 803938) Time Spent: 8h 20m (was: 8h 10m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 8h 20m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803857 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 26/Aug/22 07:19 Start Date: 26/Aug/22 07:19 Worklog Time Spent: 10m Work Description: RyanSkraba commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1228143390 Thanks for stepping up here @KyleSchoonover and @KalleOlaviNiemitalo! When you approve this PR, let's merge it for 1.12.0! Issue Time Tracking --- Worklog Id: (was: 803857) Time Spent: 8h 10m (was: 8h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 8h 10m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803795 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 21:55 Start Date: 25/Aug/22 21:55 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1227798783 Thanks for the review @KyleSchoonover ! I believe I've incorporated all your feedback. Please let me know if I missed something. I look forward to the rest of your review! Issue Time Tracking --- Worklog Id: (was: 803795) Time Spent: 8h (was: 7h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > Time Spent: 8h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803794
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 21:51
Start Date: 25/Aug/22 21:51
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955451590
##
lang/csharp/src/apache/main/IO/Parsing/Symbol.cs:
##
@@ -0,0 +1,1014 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Symbol is the base of all symbols (terminals and non-terminals) of the
+/// grammar.
+///
+public abstract class Symbol
+{
+///
+/// The type of symbol.
+///
+public enum Kind
+{
+///
+/// terminal symbols which have no productions
+Terminal,
+
+///
+/// Start symbol for some grammar
+Root,
+
+///
+/// non-terminal symbol which is a sequence of one or more other
symbols
+Sequence,
+
+///
+/// non-terminal to represent the contents of an array or map
+Repeater,
+
+///
+/// non-terminal to represent the union
+Alternative,
+
+///
+/// non-terminal action symbol which are automatically consumed
+ImplicitAction,
+
+///
+/// non-terminal action symbol which is explicitly consumed
+ExplicitAction
+}
+
+/// The kind of this symbol.
+public readonly Kind SymKind;
+
+///
+/// The production for this symbol. If this symbol is a terminal this
is
+/// null. Otherwise this holds the the sequence of the
symbols that
+/// forms the production for this symbol. The sequence is in the
reverse order of
+/// production. This is useful for easy copying onto parsing stack.
+///
+/// Please note that this is a final. So the production for a symbol
should be
+/// known before that symbol is constructed. This requirement cannot
be met for
+/// those symbols which are recursive (e.g. a record that holds union
a branch of
+/// which is the record itself). To resolve this problem, we
initialize the
+/// symbol with an array of nulls. Later we fill the symbols. Not
clean, but
+/// works. The other option is to not have this field a final. But
keeping it
+/// final and thus keeping symbol immutable gives some comfort. See
various
+/// generators how we generate records.
+///
+public readonly Symbol[] Production;
+
+///
+/// Constructs a new symbol of the given kind.
+///
+protected Symbol(Kind kind) : this(kind, null)
+{
+}
+
+///
+/// Constructs a new symbol of the given kind and production.
+///
+protected Symbol(Kind kind, Symbol[] production)
+{
+this.Production = production;
+this.SymKind = kind;
+}
+
+///
+/// A convenience method to construct a root symbol.
+///
+public static Symbol NewRoot(params Symbol[] symbols)
+{
+return new Root(symbols);
+}
+
+///
+/// A convenience method to construct a sequence.
+///
+/// The constituent symbols of the sequence.
+public static Symbol NewSeq(params Symbol[] production)
+{
+return new Sequence(production);
+}
+
+///
+/// A convenience method to construct a repeater.
+///
+/// The end symbol.
+/// The symbols to repeat in the repeater.
+public static Symbol NewRepeat(Symbol endSymbol, params Symbol[]
symsToRepeat)
+{
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803793
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 21:33
Start Date: 25/Aug/22 21:33
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955440899
##
lang/csharp/src/apache/main/IO/Parsing/Parser.cs:
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Parser is the class that maintains the stack for parsing. This class
is used
+/// by encoders, which are not required to skip.
+///
+public class Parser
+{
+///
+/// The parser knows how to handle the terminal and non-terminal
symbols. But it
+/// needs help from outside to handle implicit and explicit actions.
The clients
+/// implement this interface to provide this help.
+///
+public interface IActionHandler
+{
+///
+/// Handle the action symbol top when the input
is sought to be
+/// taken off the stack.
+///
+/// The input symbol from the caller of
Advance
+///The symbol at the top the stack.
+/// null if Advance() is to continue processing
the stack. If
+/// not null the return value will be returned by
Advance().
+Symbol DoAction(Symbol input, Symbol top);
+}
+
+private readonly IActionHandler symbolHandler;
+///
+/// Stack of symbols.
+///
+protected Symbol[] Stack;
+///
+/// Position of the stack.
+///
+protected int Pos;
+
+///
+/// Initializes a new instance of the class.
+///
+public Parser(Symbol root, IActionHandler symbolHandler)
+{
+this.symbolHandler = symbolHandler;
+this.Stack = new Symbol[5]; // Start small to make sure expansion
code works
+this.Stack[0] = root;
+this.Pos = 1;
+}
+
+///
+/// If there is no sufficient room in the stack, use this expand it.
+///
+private void expandStack()
+{
+Array.Resize(ref Stack, Stack.Length + Math.Max(Stack.Length,
1024));
+}
+
+///
+/// Recursively replaces the symbol at the top of the stack with its
production,
+/// until the top is a terminal. Then checks if the top symbol matches
the
+/// terminal symbol supplied input.
+///
+/// The symbol to match against the terminal at
the top of the
+/// stack.
+/// The terminal symbol at the top of the stack unless an
implicit action
+/// resulted in another symbol, in which case that symbol is
returned.
+public Symbol Advance(Symbol input)
+{
+for (;;)
+{
+Symbol top = Stack[--Pos];
+if (top == input)
+{
+return top; // A common case
+}
+
+Symbol.Kind k = top.SymKind;
+if (k == Symbol.Kind.ImplicitAction)
+{
+Symbol result = symbolHandler.DoAction(input, top);
+if (result != null)
+{
+return result;
+}
+}
+else if (k == Symbol.Kind.Terminal)
+{
+throw new AvroTypeException("Attempt to process a " +
input + " when a " + top + " was expected.");
+}
+else if (k == Symbol.Kind.Repeater && input ==
((Symbol.Repeater)top).End)
+{
+return input;
+}
+else
+{
+PushProduction(top);
Review Comment:
T
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803792&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803792
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 21:32
Start Date: 25/Aug/22 21:32
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955440136
##
lang/csharp/src/apache/main/IO/JsonDecoder.cs:
##
@@ -0,0 +1,765 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Text;
+using Avro.IO.Parsing;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// A for Avro's JSON data encoding.
+///
+/// JsonDecoder is not thread-safe.
+///
+public class JsonDecoder : ParsingDecoder
+{
+private JsonReader reader;
+private readonly Stack reorderBuffers = new
Stack();
+private ReorderBuffer currentReorderBuffer;
+
+private class ReorderBuffer
+{
+public readonly IDictionary>
SavedFields =
+new Dictionary>();
+
+public JsonReader OrigParser;
+}
+
+private JsonDecoder(Symbol root, Stream stream) : base(root)
+{
+Configure(stream);
+}
+
+private JsonDecoder(Symbol root, string str) : base(root)
+{
+Configure(str);
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonDecoder(Schema schema, Stream stream) :
this(getSymbol(schema), stream)
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonDecoder(Schema schema, string str) :
this(getSymbol(schema), str)
+{
+}
+
+private static Symbol getSymbol(Schema schema)
+{
+return (new JsonGrammarGenerator()).Generate(schema);
+}
+
+///
+/// Reconfigures this JsonDecoder to use the InputStream provided.
+/// Otherwise, this JsonDecoder will reset its state and then
reconfigure its
+/// input.
+///
+/// The InputStream to read from. Cannot be
null.
+/// this JsonDecoder
+public JsonDecoder Configure(Stream stream)
+{
+Parser.Reset();
+reorderBuffers.Clear();
+currentReorderBuffer = null;
+this.reader = new JsonTextReader(new StreamReader(stream));
+this.reader.Read();
+return this;
+}
+
+///
+/// Reconfigures this JsonDecoder to use the String provided for input.
+/// Otherwise, this JsonDecoder will reset its state and then
reconfigure its
+/// input.
+///
+/// The String to read from. Cannot be null.
+/// this JsonDecoder
+public JsonDecoder Configure(string str)
+{
+Parser.Reset();
+reorderBuffers.Clear();
+currentReorderBuffer = null;
+this.reader = new JsonTextReader(new StringReader(str));
+this.reader.Read();
+return this;
+}
+
+private void advance(Symbol symbol)
+{
+this.Parser.ProcessTrailingImplicitActions();
+Parser.Advance(symbol);
+}
+
+///
+public override void ReadNull()
+{
+advance(Symbol.Null);
+if (reader.TokenType == JsonToken.Null)
+{
+reader.Read();
+}
+else
+{
+throw error("null");
+}
+}
+
+///
+public override bool ReadBoolean()
+{
+advance(Symbol.Boolean);
+if (reader.TokenType == JsonToken.Boolean)
+{
+bool result = Convert.ToBoolean(reader.Value);
+reader.Read();
+return result;
+}
+else
+{
+
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803791
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 21:31
Start Date: 25/Aug/22 21:31
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955439815
##
lang/csharp/src/apache/main/IO/JsonDecoder.cs:
##
@@ -0,0 +1,765 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Text;
+using Avro.IO.Parsing;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// A for Avro's JSON data encoding.
+///
+/// JsonDecoder is not thread-safe.
+///
+public class JsonDecoder : ParsingDecoder
+{
+private JsonReader reader;
+private readonly Stack reorderBuffers = new
Stack();
+private ReorderBuffer currentReorderBuffer;
+
+private class ReorderBuffer
+{
+public readonly IDictionary>
SavedFields =
+new Dictionary>();
+
+public JsonReader OrigParser;
+}
+
+private JsonDecoder(Symbol root, Stream stream) : base(root)
+{
+Configure(stream);
+}
+
+private JsonDecoder(Symbol root, string str) : base(root)
+{
+Configure(str);
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonDecoder(Schema schema, Stream stream) :
this(getSymbol(schema), stream)
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonDecoder(Schema schema, string str) :
this(getSymbol(schema), str)
+{
+}
+
+private static Symbol getSymbol(Schema schema)
+{
+return (new JsonGrammarGenerator()).Generate(schema);
+}
+
+///
+/// Reconfigures this JsonDecoder to use the InputStream provided.
+/// Otherwise, this JsonDecoder will reset its state and then
reconfigure its
+/// input.
+///
+/// The InputStream to read from. Cannot be
null.
+/// this JsonDecoder
+public JsonDecoder Configure(Stream stream)
+{
+Parser.Reset();
+reorderBuffers.Clear();
+currentReorderBuffer = null;
+this.reader = new JsonTextReader(new StreamReader(stream));
+this.reader.Read();
+return this;
+}
+
+///
+/// Reconfigures this JsonDecoder to use the String provided for input.
+/// Otherwise, this JsonDecoder will reset its state and then
reconfigure its
+/// input.
+///
+/// The String to read from. Cannot be null.
+/// this JsonDecoder
+public JsonDecoder Configure(string str)
+{
+Parser.Reset();
+reorderBuffers.Clear();
+currentReorderBuffer = null;
+this.reader = new JsonTextReader(new StringReader(str));
+this.reader.Read();
+return this;
+}
+
+private void advance(Symbol symbol)
+{
+this.Parser.ProcessTrailingImplicitActions();
+Parser.Advance(symbol);
+}
+
+///
+public override void ReadNull()
+{
+advance(Symbol.Null);
+if (reader.TokenType == JsonToken.Null)
+{
+reader.Read();
+}
+else
+{
+throw error("null");
+}
+}
+
+///
+public override bool ReadBoolean()
+{
+advance(Symbol.Boolean);
+if (reader.TokenType == JsonToken.Boolean)
+{
+bool result = Convert.ToBoolean(reader.Value);
+reader.Read();
+return result;
+}
+else
+{
+
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803786&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803786
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 21:02
Start Date: 25/Aug/22 21:02
Worklog Time Spent: 10m
Work Description: KyleSchoonover commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955394370
##
lang/csharp/src/apache/main/IO/Parsing/Parser.cs:
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Parser is the class that maintains the stack for parsing. This class
is used
+/// by encoders, which are not required to skip.
+///
+public class Parser
+{
+///
+/// The parser knows how to handle the terminal and non-terminal
symbols. But it
+/// needs help from outside to handle implicit and explicit actions.
The clients
+/// implement this interface to provide this help.
+///
+public interface IActionHandler
+{
+///
+/// Handle the action symbol top when the input
is sought to be
+/// taken off the stack.
+///
+/// The input symbol from the caller of
Advance
+///The symbol at the top the stack.
+/// null if Advance() is to continue processing
the stack. If
+/// not null the return value will be returned by
Advance().
+Symbol DoAction(Symbol input, Symbol top);
+}
+
+private readonly IActionHandler symbolHandler;
+///
+/// Stack of symbols.
+///
+protected Symbol[] Stack;
+///
+/// Position of the stack.
+///
+protected int Pos;
+
+///
+/// Initializes a new instance of the class.
+///
+public Parser(Symbol root, IActionHandler symbolHandler)
+{
+this.symbolHandler = symbolHandler;
+this.Stack = new Symbol[5]; // Start small to make sure expansion
code works
+this.Stack[0] = root;
+this.Pos = 1;
+}
+
+///
+/// If there is no sufficient room in the stack, use this expand it.
+///
+private void expandStack()
+{
+Array.Resize(ref Stack, Stack.Length + Math.Max(Stack.Length,
1024));
+}
+
+///
+/// Recursively replaces the symbol at the top of the stack with its
production,
+/// until the top is a terminal. Then checks if the top symbol matches
the
+/// terminal symbol supplied input.
+///
+/// The symbol to match against the terminal at
the top of the
+/// stack.
+/// The terminal symbol at the top of the stack unless an
implicit action
+/// resulted in another symbol, in which case that symbol is
returned.
+public Symbol Advance(Symbol input)
+{
+for (;;)
+{
+Symbol top = Stack[--Pos];
+if (top == input)
+{
+return top; // A common case
+}
+
+Symbol.Kind k = top.SymKind;
+if (k == Symbol.Kind.ImplicitAction)
+{
+Symbol result = symbolHandler.DoAction(input, top);
+if (result != null)
+{
+return result;
+}
+}
+else if (k == Symbol.Kind.Terminal)
+{
+throw new AvroTypeException("Attempt to process a " +
input + " when a " + top + " was expected.");
+}
+else if (k == Symbol.Kind.Repeater && input ==
((Symbol.Repeater)top).End)
+{
+return input;
+}
+else
+{
+PushProduction(top);
Review Comment
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803764
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 20:02
Start Date: 25/Aug/22 20:02
Worklog Time Spent: 10m
Work Description: KyleSchoonover commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955370460
##
lang/csharp/src/apache/main/IO/JsonDecoder.cs:
##
@@ -0,0 +1,765 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Text;
+using Avro.IO.Parsing;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// A for Avro's JSON data encoding.
+///
+/// JsonDecoder is not thread-safe.
+///
+public class JsonDecoder : ParsingDecoder
+{
+private JsonReader reader;
+private readonly Stack reorderBuffers = new
Stack();
+private ReorderBuffer currentReorderBuffer;
+
+private class ReorderBuffer
+{
+public readonly IDictionary>
SavedFields =
+new Dictionary>();
+
+public JsonReader OrigParser;
Review Comment:
I am aware it is a private class. The shorter method is fine.
Issue Time Tracking
---
Worklog Id: (was: 803764)
Time Spent: 7h (was: 6h 50m)
> JsonEncode Decode support for C#
>
>
> Key: AVRO-3001
> URL: https://issues.apache.org/jira/browse/AVRO-3001
> Project: Apache Avro
> Issue Type: Improvement
> Components: csharp
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Krishnan Unni
>Assignee: Robert Yokota
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> The C# library for avro currently supports only the Binary encoding and also
> with compile time types (Generic support only). As part of a project I am
> doing I need to validate the avro schema against the incoming json data on
> the fly without a predefined type (generated class). So basically comparing
> an avro schema (string/json representation) against a raw json string. It is
> possible with the Java library since it supports both non generic types and
> streams as well as json encoding. With C# currently this is not possible. Is
> there a plan to extend the C# library to provide these features? If yes, is
> there a timeline? If not is there any alternative to achieve this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803722&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803722
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 25/Aug/22 17:58
Start Date: 25/Aug/22 17:58
Worklog Time Spent: 10m
Work Description: KalleOlaviNiemitalo commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r955266867
##
lang/csharp/src/apache/main/IO/JsonDecoder.cs:
##
@@ -0,0 +1,765 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Text;
+using Avro.IO.Parsing;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// A for Avro's JSON data encoding.
+///
+/// JsonDecoder is not thread-safe.
+///
+public class JsonDecoder : ParsingDecoder
+{
+private JsonReader reader;
+private readonly Stack reorderBuffers = new
Stack();
+private ReorderBuffer currentReorderBuffer;
+
+private class ReorderBuffer
+{
+public readonly IDictionary>
SavedFields =
+new Dictionary>();
+
+public JsonReader OrigParser;
Review Comment:
Note it is in a private class.
If you want to make it a property anyway, I think `public JsonReader
OrigParser { get; set; }` would be simpler.
Issue Time Tracking
---
Worklog Id: (was: 803722)
Time Spent: 6h 50m (was: 6h 40m)
> JsonEncode Decode support for C#
>
>
> Key: AVRO-3001
> URL: https://issues.apache.org/jira/browse/AVRO-3001
> Project: Apache Avro
> Issue Type: Improvement
> Components: csharp
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Krishnan Unni
>Assignee: Robert Yokota
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> The C# library for avro currently supports only the Binary encoding and also
> with compile time types (Generic support only). As part of a project I am
> doing I need to validate the avro schema against the incoming json data on
> the fly without a predefined type (generated class). So basically comparing
> an avro schema (string/json representation) against a raw json string. It is
> possible with the Java library since it supports both non generic types and
> streams as well as json encoding. With C# currently this is not possible. Is
> there a plan to extend the C# library to provide these features? If yes, is
> there a timeline? If not is there any alternative to achieve this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803656 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 14:04 Start Date: 25/Aug/22 14:04 Worklog Time Spent: 10m Work Description: rayokota commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r955011643 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: Thanks @KalleOlaviNiemitalo Issue Time Tracking --- Worklog Id: (was: 803656) Time Spent: 6.5h (was: 6h 20m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 6.5h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803494&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803494 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 07:51 Start Date: 25/Aug/22 07:51 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r954622532 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: JsonEncoder.Flush would be OK for branch-1.11, of course. Only Encoder.Flush is a compatibility problem. I see Avro.IO.BinaryEncoder already has `public void Flush()` in release-1.11.1. Issue Time Tracking --- Worklog Id: (was: 803494) Time Spent: 6h 20m (was: 6h 10m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 6h 20m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803488&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803488 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 07:37 Start Date: 25/Aug/22 07:37 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r954605087 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: No, I think it is OK to keep. But if you need the JSON encoder in branch-1.11, then Flush should be omitted from that PR. Issue Time Tracking --- Worklog Id: (was: 803488) Time Spent: 6h 10m (was: 6h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 6h 10m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803473 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 07:19 Start Date: 25/Aug/22 07:19 Worklog Time Spent: 10m Work Description: rayokota commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r954589056 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: Thanks @KalleOlaviNiemitalo , would you prefer that I remove this? Issue Time Tracking --- Worklog Id: (was: 803473) Time Spent: 6h (was: 5h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 6h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803470&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803470 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 07:18 Start Date: 25/Aug/22 07:18 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1226872623 > My C# knowledge is too weak to be able to review such big PR :-/ > I'll let the C# devs to manage this one. Thanks @martin-g . This PR is mostly a line-for-line port of the following Java classes: - JsonGrammarGenerator.java - Parser.java - SkipParser.java - Symbol.java - ValidatingGrammarGenerator.java - ParsingEncoder.java - ParsingDecoder.java - JsonEncoder.java - JsonDecoder.java In any case, who are the C# devs who would be able to merge this? Issue Time Tracking --- Worklog Id: (was: 803470) Time Spent: 5h 50m (was: 5h 40m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803458&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803458 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 06:40 Start Date: 25/Aug/22 06:40 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on code in PR #1833: URL: https://github.com/apache/avro/pull/1833#discussion_r954557727 ## lang/csharp/src/apache/main/IO/Encoder.cs: ## @@ -187,5 +187,10 @@ public interface Encoder /// Position within data where the contents start. /// Number of bytes to write. void WriteFixed(byte[] data, int start, int len); + +/// +/// Flushes the encoder. +/// +void Flush(); Review Comment: Adding a method to a public interface is a breaking change. It is OK for this pull request to master (upcoming 1.12.0) but I think these changes should not be backported to branch-1.11. Issue Time Tracking --- Worklog Id: (was: 803458) Time Spent: 5h 40m (was: 5.5h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803456&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803456 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 25/Aug/22 06:35 Start Date: 25/Aug/22 06:35 Worklog Time Spent: 10m Work Description: martin-g commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1226836507 My C# knowledge is too weak to be able to review such big PR :-/ I'll let the C# devs to manage this one. Issue Time Tracking --- Worklog Id: (was: 803456) Time Spent: 5.5h (was: 5h 20m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 5.5h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803244&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803244 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 24/Aug/22 14:33 Start Date: 24/Aug/22 14:33 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1225811180 Thanks for the approval @KalleOlaviNiemitalo ! @martin-g @KyleSchoonover @zcsizmadia Could one of you kindly merge this please? I would greatly appreciate it! Issue Time Tracking --- Worklog Id: (was: 803244) Time Spent: 5h 20m (was: 5h 10m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803163&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803163 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 24/Aug/22 08:38 Start Date: 24/Aug/22 08:38 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1225380917 > Ok so I should file a separate issue for name vs. fullname in JSON encoding of unions. Filed as [AVRO-3620](https://issues.apache.org/jira/browse/AVRO-3620). Issue Time Tracking --- Worklog Id: (was: 803163) Time Spent: 5h 10m (was: 5h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803103
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 24/Aug/22 04:47
Start Date: 24/Aug/22 04:47
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r953339960
##
lang/csharp/src/apache/main/IO/Parsing/Symbol.cs:
##
@@ -0,0 +1,1049 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Symbol is the base of all symbols (terminals and non-terminals) of the
+/// grammar.
+///
+public abstract class Symbol
+{
+///
+/// The type of symbol.
+///
+public enum Kind
+{
+///
+/// terminal symbols which have no productions
+Terminal,
+
+///
+/// Start symbol for some grammar
+Root,
+
+///
+/// non-terminal symbol which is a sequence of one or more other
symbols
+Sequence,
+
+///
+/// non-terminal to represent the contents of an array or map
+Repeater,
+
+///
+/// non-terminal to represent the union
+Alternative,
+
+///
+/// non-terminal action symbol which are automatically consumed
+ImplicitAction,
+
+///
+/// non-terminal action symbol which is explicitly consumed
+ExplicitAction
+}
+
+/// The kind of this symbol.
+public readonly Kind SymKind;
+
+///
+/// The production for this symbol. If this symbol is a terminal this
is
+/// null. Otherwise this holds the the sequence of the
symbols that
+/// forms the production for this symbol. The sequence is in the
reverse order of
+/// production. This is useful for easy copying onto parsing stack.
+///
+/// Please note that this is a final. So the production for a symbol
should be
+/// known before that symbol is constructed. This requirement cannot
be met for
+/// those symbols which are recursive (e.g. a record that holds union
a branch of
+/// which is the record itself). To resolve this problem, we
initialize the
+/// symbol with an array of nulls. Later we fill the symbols. Not
clean, but
+/// works. The other option is to not have this field a final. But
keeping it
+/// final and thus keeping symbol immutable gives some comfort. See
various
+/// generators how we generate records.
+///
+public readonly Symbol[] Production;
+
+///
+/// Constructs a new symbol of the given kind.
+///
+protected Symbol(Kind kind) : this(kind, null)
+{
+}
+
+///
+/// Constructs a new symbol of the given kind and production.
+///
+protected Symbol(Kind kind, Symbol[] production)
+{
+this.Production = production;
+this.SymKind = kind;
+}
+
+///
+/// A convenience method to construct a root symbol.
+///
+public static Symbol NewRoot(params Symbol[] symbols)
+{
+return new Root(symbols);
+}
+
+///
+/// A convenience method to construct a sequence.
+///
+/// The constituent symbols of the sequence.
+public static Symbol NewSeq(params Symbol[] production)
+{
+return new Sequence(production);
+}
+
+///
+/// A convenience method to construct a repeater.
+///
+/// The end symbol.
+/// The symbols to repeat in the repeater.
+public static Symbol NewRepeat(Symbol endSymbol, params Symbol[]
symsToRepeat)
+{
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803105 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 24/Aug/22 04:47 Start Date: 24/Aug/22 04:47 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1225186562 @KalleOlaviNiemitalo , I've incorporated your latest review feedback, thanks for your thoroughness. Issue Time Tracking --- Worklog Id: (was: 803105) Time Spent: 5h (was: 4h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803104
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 24/Aug/22 04:47
Start Date: 24/Aug/22 04:47
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r953340246
##
lang/csharp/src/apache/main/IO/Parsing/Parser.cs:
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Parser is the class that maintains the stack for parsing. This class
is used
+/// by encoders, which are not required to skip.
+///
+public class Parser
+{
+///
+/// The parser knows how to handle the terminal and non-terminal
symbols. But it
+/// needs help from outside to handle implicit and explicit actions.
The clients
+/// implement this interface to provide this help.
+///
+public interface IActionHandler
+{
+///
+/// Handle the action symbol top when the input
is sought to be
+/// taken off the stack.
+///
+/// The input symbol from the caller of
advance
+///The symbol at the top the stack.
+/// null if advance() is to continue processing
the stack. If
+/// not null the return value will be returned by
advance().
Review Comment:
Done
Issue Time Tracking
---
Worklog Id: (was: 803104)
Time Spent: 4h 50m (was: 4h 40m)
> JsonEncode Decode support for C#
>
>
> Key: AVRO-3001
> URL: https://issues.apache.org/jira/browse/AVRO-3001
> Project: Apache Avro
> Issue Type: Improvement
> Components: csharp
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Krishnan Unni
>Assignee: Robert Yokota
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> The C# library for avro currently supports only the Binary encoding and also
> with compile time types (Generic support only). As part of a project I am
> doing I need to validate the avro schema against the incoming json data on
> the fly without a predefined type (generated class). So basically comparing
> an avro schema (string/json representation) against a raw json string. It is
> possible with the Java library since it supports both non generic types and
> streams as well as json encoding. With C# currently this is not possible. Is
> there a plan to extend the C# library to provide these features? If yes, is
> there a timeline? If not is there any alternative to achieve this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803099
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 24/Aug/22 04:45
Start Date: 24/Aug/22 04:45
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r953338257
##
lang/csharp/src/apache/main/IO/Parsing/Parser.cs:
##
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+
+namespace Avro.IO.Parsing
+{
+///
+/// Parser is the class that maintains the stack for parsing. This class
is used
+/// by encoders, which are not required to skip.
+///
+public class Parser
+{
+///
+/// The parser knows how to handle the terminal and non-terminal
symbols. But it
+/// needs help from outside to handle implicit and explicit actions.
The clients
+/// implement this interface to provide this help.
+///
+public interface IActionHandler
+{
+///
+/// Handle the action symbol top when the input
is sought to be
+/// taken off the stack.
+///
+/// The input symbol from the caller of
advance
+///The symbol at the top the stack.
+/// null if advance() is to continue processing
the stack. If
+/// not null the return value will be returned by
advance().
+Symbol DoAction(Symbol input, Symbol top);
+}
+
+private readonly IActionHandler symbolHandler;
+///
+/// Stack of symbols.
+///
+protected Symbol[] Stack;
+///
+/// Position of the stack.
+///
+protected int Pos;
+
+///
+/// Initializes a new instance of the class.
+///
+public Parser(Symbol root, IActionHandler symbolHandler)
+{
+this.symbolHandler = symbolHandler;
+this.Stack = new Symbol[5]; // Start small to make sure expansion
code works
+this.Stack[0] = root;
+this.Pos = 1;
+}
+
+///
+/// If there is no sufficient room in the stack, use this expand it.
+///
+private void expandStack()
+{
+Array.Resize(ref Stack, Stack.Length + Math.Max(Stack.Length,
1024));
+}
+
+///
+/// Recursively replaces the symbol at the top of the stack with its
production,
+/// until the top is a terminal. Then checks if the top symbol matches
the
+/// terminal symbol supplied terminal.
Review Comment:
Done
##
lang/csharp/src/apache/main/IO/Parsing/SkipParser.cs:
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System.Diagnostics;
+
+namespace Avro.IO.Parsing
+{
+///
+/// A parser that capable of skipping as well read and write. This class
is used
+/// by decoders who (unlink encoders) are required to implement methods to
skip.
Review Comment:
Done
Issue Time Tracking
---
Worklog Id: (was: 803099)
Time Spent: 4.5h (was: 4h 20m)
> JsonEncode Decode support for C#
> --
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803098
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 24/Aug/22 04:44
Start Date: 24/Aug/22 04:44
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r953338043
##
lang/csharp/src/apache/main/IO/JsonEncoder.cs:
##
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using Avro.IO.Parsing;
+using System.Collections;
+using System.IO;
+using System.Text;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// An for Avro's JSON data encoding.
+///
+/// JsonEncoder buffers output, and data may not appear on the output until
+/// is called.
+///
+/// JsonEncoder is not thread-safe.
+///
+public class JsonEncoder : ParsingEncoder, Parser.IActionHandler
+{
+private readonly Parser parser;
+private JsonWriter writer;
+private bool includeNamespace = true;
+
+// Has anything been written into the collections?
+private readonly BitArray isEmpty = new BitArray(64);
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, Stream stream) : this(sc,
getJsonWriter(stream, false))
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, Stream stream, bool pretty) : this(sc,
getJsonWriter(stream, pretty))
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, JsonWriter writer)
+{
+Configure(writer);
+this.parser = new Parser((new
JsonGrammarGenerator()).Generate(sc), this);
+}
+
+///
+public override void Flush()
+{
+parser.ProcessImplicitActions();
+if (writer != null)
+{
+writer.Flush();
+}
+}
+
+// by default, one object per line.
+// with pretty option use default pretty printer with root line
separator.
+private static JsonWriter getJsonWriter(Stream stream, bool pretty)
+{
+JsonWriter writer = new JsonTextWriter(new StreamWriter(stream));
+if (pretty)
+{
+writer.Formatting = Formatting.Indented;
+}
+
+return writer;
+}
+
+///
+/// Whether to include the namespace.
+///
+public virtual bool IncludeNamespace
Review Comment:
Done
##
lang/csharp/src/apache/main/IO/Parsing/JsonGrammarGenerator.cs:
##
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections.Generic;
+
+namespace Avro.IO.Parsing
+{
+///
+/// The class that generates a grammar suitable to parse Avro data in JSON
+/// format.
+///
+public class JsonGrammarGenerator : ValidatingGrammarGenerator
+{
+///
+/// Returns the non-terminal that is the start symbol for the grammar
for the
+/// grammar for t
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803092&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803092
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 24/Aug/22 04:21
Start Date: 24/Aug/22 04:21
Worklog Time Spent: 10m
Work Description: KalleOlaviNiemitalo commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r953303802
##
lang/csharp/src/apache/main/IO/Parsing/JsonGrammarGenerator.cs:
##
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using System;
+using System.Collections.Generic;
+
+namespace Avro.IO.Parsing
+{
+///
+/// The class that generates a grammar suitable to parse Avro data in JSON
+/// format.
+///
+public class JsonGrammarGenerator : ValidatingGrammarGenerator
+{
+///
+/// Returns the non-terminal that is the start symbol for the grammar
for the
+/// grammar for the given schema sc.
Review Comment:
The parameter is named `schema`, not `sc`.
##
lang/csharp/src/apache/main/IO/JsonEncoder.cs:
##
@@ -0,0 +1,356 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using Avro.IO.Parsing;
+using System.Collections;
+using System.IO;
+using System.Text;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// An for Avro's JSON data encoding.
+///
+/// JsonEncoder buffers output, and data may not appear on the output until
+/// is called.
+///
+/// JsonEncoder is not thread-safe.
+///
+public class JsonEncoder : ParsingEncoder, Parser.IActionHandler
+{
+private readonly Parser parser;
+private JsonWriter writer;
+private bool includeNamespace = true;
+
+// Has anything been written into the collections?
+private readonly BitArray isEmpty = new BitArray(64);
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, Stream stream) : this(sc,
getJsonWriter(stream, false))
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, Stream stream, bool pretty) : this(sc,
getJsonWriter(stream, pretty))
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, JsonWriter writer)
+{
+Configure(writer);
+this.parser = new Parser((new
JsonGrammarGenerator()).Generate(sc), this);
+}
+
+///
+public override void Flush()
+{
+parser.ProcessImplicitActions();
+if (writer != null)
+{
+writer.Flush();
+}
+}
+
+// by default, one object per line.
+// with pretty option use default pretty printer with root line
separator.
+private static JsonWriter getJsonWriter(Stream stream, bool pretty)
+{
+JsonWriter writer = new JsonTextWriter(new StreamWriter(stream));
+if (pretty)
+{
+writer.Formatting = Formatting.Indented;
+}
+
+return writer;
+}
+
+///
+/// Whether to include the na
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803088 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 24/Aug/22 03:34 Start Date: 24/Aug/22 03:34 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1225140325 Ok so I should file a separate issue for name vs. fullname in JSON encoding of unions. Not sure which one is better but surely the spec and the implementations should be consistent with each other. Issue Time Tracking --- Worklog Id: (was: 803088) Time Spent: 4h (was: 3h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803077&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803077 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 24/Aug/22 02:01 Start Date: 24/Aug/22 02:01 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1225084782 > I see JsonCodecTests.cs has tests with true and false as JsonEncoder.IncludeNamespace; but I don't see any namespaces in the schemas that the tests use. The `includeNamespace` is just used to determine whether to emit the union label (its name is somewhat misleading). The Java code is [here](https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonEncoder.java#L297.). Since the C# is a straight port, the corresponding code is [here](https://github.com/apache/avro/blob/1133acda686fb1ff96fcbb80ca606c20e261f343/lang/csharp/src/apache/main/IO/JsonEncoder.cs#L320). Issue Time Tracking --- Worklog Id: (was: 803077) Time Spent: 3h 50m (was: 3h 40m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=803075&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803075 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 24/Aug/22 01:58 Start Date: 24/Aug/22 01:58 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1225083274 > How about a union with a member that is a record with a namespace? The [JSON Encoding](https://avro.apache.org/docs/1.11.1/specification/#json-encoding) specification says > > > otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used. > > which refers to the record _name_ rather than _fullname_. @KalleOlaviNiemitalo , I added a test for a union containing a record with a namespace as you requested. The Java code uses the fullname for the union label [here](https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/parsing/ValidatingGrammarGenerator.java#L104). Since the C# code is a straight port of the Java code, it uses the fullname for the union label [here](https://github.com/apache/avro/blob/1133acda686fb1ff96fcbb80ca606c20e261f343/lang/csharp/src/apache/main/IO/Parsing/ValidatingGrammarGenerator.cs#L111) Issue Time Tracking --- Worklog Id: (was: 803075) Time Spent: 3h 40m (was: 3.5h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802700&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802700 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 23/Aug/22 05:44 Start Date: 23/Aug/22 05:44 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1223575367 > > I've fixed that issue in this PR by overriding the `Fullname` property in `LogicalSchema`. > > That might fix [AVRO-3568](https://issues.apache.org/jira/browse/AVRO-3568) as well. Thanks @KalleOlaviNiemitalo , I was able to fix [AVRO-3568](https://issues.apache.org/jira/browse/AVRO-3568) by overriding the `Name` property in `LogicalType` as well. I'll try to add some more tests as you requested when I get a chance. Issue Time Tracking --- Worklog Id: (was: 802700) Time Spent: 3.5h (was: 3h 20m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802695&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802695 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 23/Aug/22 05:17 Start Date: 23/Aug/22 05:17 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1223556806 I see JsonCodecTests.cs has tests with true and false as JsonEncoder.IncludeNamespace; but I don't see any namespaces in the schemas that the tests use. Issue Time Tracking --- Worklog Id: (was: 802695) Time Spent: 3h 20m (was: 3h 10m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802694 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 23/Aug/22 05:12 Start Date: 23/Aug/22 05:12 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1223553494 How about a union with a member that is a record with a namespace? The [JSON Encoding](https://avro.apache.org/docs/1.11.1/specification/#json-encoding) specification says > otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used. which refers to the record _name_ rather than _fullname_. Issue Time Tracking --- Worklog Id: (was: 802694) Time Spent: 3h 10m (was: 3h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802690 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 23/Aug/22 04:43 Start Date: 23/Aug/22 04:43 Worklog Time Spent: 10m Work Description: KalleOlaviNiemitalo commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1223533178 > I've fixed that issue in this PR by overriding the `Fullname` property in `LogicalSchema`. That might fix [AVRO-3568](https://issues.apache.org/jira/browse/AVRO-3568) as well. Issue Time Tracking --- Worklog Id: (was: 802690) Time Spent: 3h (was: 2h 50m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802639 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 22/Aug/22 21:56 Start Date: 22/Aug/22 21:56 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1223162854 @KalleOlaviNiemitalo , it turns out that the schema above was not parseable in the C# implementation, due to https://issues.apache.org/jira/browse/AVRO-3613. I've fixed that issue in this PR by overriding the `Fullname` property in `LogicalSchema`. Fixing that also fixed some behavior in the `JsonGrammarGenerator` and the `ValidatingGrammarGenerator` that relied on `Fullname`. There was another issue with your schema in that there was an ambiguity when trying to JSON encode a `Date`, since both logicalTypes could be used to represent a `Date`. I changed your example to have one of the fields be a logical type of `Uuid` instead. Issue Time Tracking --- Worklog Id: (was: 802639) Time Spent: 2h 50m (was: 2h 40m) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[ https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802602&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802602 ] ASF GitHub Bot logged work on AVRO-3001: Author: ASF GitHub Bot Created on: 22/Aug/22 19:25 Start Date: 22/Aug/22 19:25 Worklog Time Spent: 10m Work Description: rayokota commented on PR #1833: URL: https://github.com/apache/avro/pull/1833#issuecomment-1222815474 @KalleOlaviNiemitalo , actually the schema above does not parse in C#. It turns out that the C# implementation only allows one logical type in a union. I've filed https://issues.apache.org/jira/browse/AVRO-3613. In any case, I've added a test with one logical type in a union. Please let me know if there's anything else you'd like to see. Thanks! Issue Time Tracking --- Worklog Id: (was: 802602) Time Spent: 2h 40m (was: 2.5h) > JsonEncode Decode support for C# > > > Key: AVRO-3001 > URL: https://issues.apache.org/jira/browse/AVRO-3001 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Affects Versions: 1.10.0, 1.11.0 >Reporter: Krishnan Unni >Assignee: Robert Yokota >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > The C# library for avro currently supports only the Binary encoding and also > with compile time types (Generic support only). As part of a project I am > doing I need to validate the avro schema against the incoming json data on > the fly without a predefined type (generated class). So basically comparing > an avro schema (string/json representation) against a raw json string. It is > possible with the Java library since it supports both non generic types and > streams as well as json encoding. With C# currently this is not possible. Is > there a plan to extend the C# library to provide these features? If yes, is > there a timeline? If not is there any alternative to achieve this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (AVRO-3001) JsonEncode Decode support for C#
[
https://issues.apache.org/jira/browse/AVRO-3001?focusedWorklogId=802496&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802496
]
ASF GitHub Bot logged work on AVRO-3001:
Author: ASF GitHub Bot
Created on: 22/Aug/22 14:50
Start Date: 22/Aug/22 14:50
Worklog Time Spent: 10m
Work Description: rayokota commented on code in PR #1833:
URL: https://github.com/apache/avro/pull/1833#discussion_r951537746
##
lang/csharp/src/apache/main/IO/JsonEncoder.cs:
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+using Avro.IO.Parsing;
+using System.Collections;
+using System.IO;
+using System.Text;
+using Newtonsoft.Json;
+
+namespace Avro.IO
+{
+///
+/// An for Avro's JSON data encoding.
+///
+/// JsonEncoder buffers output, and data may not appear on the output until
+/// is called.
+///
+/// JsonEncoder is not thread-safe.
+///
+public class JsonEncoder : ParsingEncoder, Parser.ActionHandler
+{
+private readonly Parser parser;
+private JsonWriter writer;
+private bool includeNamespace = true;
+
+// Has anything been written into the collections?
+private readonly BitArray isEmpty = new BitArray(64);
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, Stream stream) : this(sc,
getJsonWriter(stream, false))
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, Stream stream, bool pretty) : this(sc,
getJsonWriter(stream, pretty))
+{
+}
+
+///
+/// Initializes a new instance of the class.
+///
+public JsonEncoder(Schema sc, JsonWriter writer)
+{
+Configure(writer);
+this.parser = new Parser((new
JsonGrammarGenerator()).Generate(sc), this);
+}
+
+///
+public override void Flush()
+{
+parser.ProcessImplicitActions();
+if (writer != null)
+{
+writer.Flush();
+}
+}
+
+// by default, one object per line.
+// with pretty option use default pretty printer with root line
separator.
+private static JsonWriter getJsonWriter(Stream stream, bool pretty)
+{
+JsonWriter writer = new JsonTextWriter(new StreamWriter(stream));
+if (pretty)
+{
+writer.Formatting = Formatting.Indented;
+}
+
+return writer;
+}
+
+///
+/// Whether to include the namespace.
+///
+public virtual bool IncludeNamespace
+{
+get { return includeNamespace; }
+set { this.includeNamespace = value; }
+}
+
+
+///
+/// Reconfigures this JsonEncoder to use the output stream provided.
+///
+/// Otherwise, this JsonEncoder will flush its current output and then
+/// reconfigure its output to use a default UTF8 JsonWriter that
writes to the
+/// provided OutputStream.
+///
+/// The OutputStream to direct output to. Cannot
be null.
+/// this JsonEncoder
+public JsonEncoder Configure(Stream stream)
+{
+this.Configure(getJsonWriter(stream, false));
+return this;
+}
+
+///
+/// Reconfigures this JsonEncoder to output to the JsonWriter provided.
+///
+/// Otherwise, this JsonEncoder will flush its current output and then
+/// reconfigure its output to use the provided JsonWriter.
+///
+/// The JsonWriter to direct output to.
Cannot be null.
+/// this JsonEncoder
+public JsonEncoder Configure(JsonWriter jsonWriter)
+{
+if (null != parser)
+{
+Flush();
+}
+
+this.writer = jsonWriter;
+return this;
+}
+
+///
+public overrid
