[jira] [Updated] (SPARK-46627) Streaming UI hover-over shows incorrect value
[
https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-46627:
---
Issue Type: Bug (was: Task)
> Streaming UI hover-over shows incorrect value
> -
>
> Key: SPARK-46627
> URL: https://issues.apache.org/jira/browse/SPARK-46627
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming, UI, Web UI
>Affects Versions: 4.0.0
>Reporter: Wei Liu
>Assignee: Kent Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: Screenshot 2024-01-08 at 1.55.57 PM.png, Screenshot
> 2024-01-08 at 15.06.24.png
>
>
> Running a simple streaming query:
> val df = spark.readStream.format("rate").option("rowsPerSecond",
> "5000").load()
> val q = df.writeStream.format("noop").start()
>
> The hover-over value is incorrect in the streaming ui (shows 321.00 at
> undefined)
>
> !Screenshot 2024-01-08 at 1.55.57 PM.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46627) Streaming UI hover-over shows incorrect value
[
https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-46627.
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved in https://github.com/apache/spark/pull/44633
> Streaming UI hover-over shows incorrect value
> -
>
> Key: SPARK-46627
> URL: https://issues.apache.org/jira/browse/SPARK-46627
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming, UI, Web UI
>Affects Versions: 4.0.0
>Reporter: Wei Liu
>Assignee: Kent Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: Screenshot 2024-01-08 at 1.55.57 PM.png, Screenshot
> 2024-01-08 at 15.06.24.png
>
>
> Running a simple streaming query:
> val df = spark.readStream.format("rate").option("rowsPerSecond",
> "5000").load()
> val q = df.writeStream.format("noop").start()
>
> The hover-over value is incorrect in the streaming ui (shows 321.00 at
> undefined)
>
> !Screenshot 2024-01-08 at 1.55.57 PM.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46627) Streaming UI hover-over shows incorrect value
[
https://issues.apache.org/jira/browse/SPARK-46627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta reassigned SPARK-46627:
--
Assignee: Kent Yao
> Streaming UI hover-over shows incorrect value
> -
>
> Key: SPARK-46627
> URL: https://issues.apache.org/jira/browse/SPARK-46627
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming, UI, Web UI
>Affects Versions: 4.0.0
>Reporter: Wei Liu
>Assignee: Kent Yao
>Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot 2024-01-08 at 1.55.57 PM.png, Screenshot
> 2024-01-08 at 15.06.24.png
>
>
> Running a simple streaming query:
> val df = spark.readStream.format("rate").option("rowsPerSecond",
> "5000").load()
> val q = df.writeStream.format("noop").start()
>
> The hover-over value is incorrect in the streaming ui (shows 321.00 at
> undefined)
>
> !Screenshot 2024-01-08 at 1.55.57 PM.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (AVRO-3897) Disallow invalid namespace in fully qualified name for Rust SDK
[
https://issues.apache.org/jira/browse/AVRO-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3897:
-
Description:
Currently, the Rust SDK allows the following fully qualified names with
Name::new.
{code}
Name::new("ns.0.record1")
Name::new("ns..record1")
{code}
But they should be disallowed according to the specification.
https://avro.apache.org/docs/1.11.1/specification/#names
{code}
The name portion of the fullname of named types, record field names, and enum
symbols must:
start with [A-Za-z_]
subsequently contain only [A-Za-z0-9_]
{code}
{code}
The null namespace may not be used in a dot-separated sequence of names. So the
grammar for a namespace is:
| [()*]
{code}
was:
Currently, the Rust SDK allows the following fully qualified names with
Name::new.
{code}
Name::new("ns.0.record1")
Name::new("ns..record1")
{code}
But they should be disallowed according to the specification.
> Disallow invalid namespace in fully qualified name for Rust SDK
> ---
>
> Key: AVRO-3897
> URL: https://issues.apache.org/jira/browse/AVRO-3897
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Reporter: Kousuke Saruta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, the Rust SDK allows the following fully qualified names with
> Name::new.
> {code}
> Name::new("ns.0.record1")
> Name::new("ns..record1")
> {code}
> But they should be disallowed according to the specification.
> https://avro.apache.org/docs/1.11.1/specification/#names
> {code}
> The name portion of the fullname of named types, record field names, and enum
> symbols must:
> start with [A-Za-z_]
> subsequently contain only [A-Za-z0-9_]
> {code}
> {code}
> The null namespace may not be used in a dot-separated sequence of names. So
> the grammar for a namespace is:
>| [()*]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3897) Disallow invalid namespace in fully qualified name for Rust SDK
Kousuke Saruta created AVRO-3897:
Summary: Disallow invalid namespace in fully qualified name for
Rust SDK
Key: AVRO-3897
URL: https://issues.apache.org/jira/browse/AVRO-3897
Project: Apache Avro
Issue Type: Bug
Components: rust
Reporter: Kousuke Saruta
Currently, the Rust SDK allows the following fully qualified names with
Name::new.
{code}
Name::new("ns.0.record1")
Name::new("ns..record1")
{code}
But they should be disallowed according to the specification.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3880) Upgrade maven-antrun-plugin to 3.1.0
Kousuke Saruta created AVRO-3880:
Summary: Upgrade maven-antrun-plugin to 3.1.0
Key: AVRO-3880
URL: https://issues.apache.org/jira/browse/AVRO-3880
Project: Apache Avro
Issue Type: Improvement
Components: build, java
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
In the current main branch, if we try to generate .class files from .proto
files for test, we will get an error like as follows.
{code}
$ mvn -P protobuf-generate generate-test-sources
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run (default) on project
avro-protobuf: You are using 'tasks' which has been removed from the
maven-antrun-plugin. Please use 'target' and refer to the >>Major Version
Upgrade to version 3.0.0<< on the plugin site. -> [Help 1]
{code}
This is because maven-antrun-plugin 3.1.0 is automatically downloaded but
"tasks" tag is not supported by the version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3870) Speed up CI for Rust
Kousuke Saruta created AVRO-3870: Summary: Speed up CI for Rust Key: AVRO-3870 URL: https://issues.apache.org/jira/browse/AVRO-3870 Project: Apache Avro Issue Type: Improvement Components: build, rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta In the current master, there are something wrong about actions/cache in `test-lang-rust-ci.yml`. First, a directory "target" is tend to be cached but the path is wrong. the correct path is "lang/rust/target", not "~/target". Second, as of Rust 1.70.0, Cargo changes the way to download dependencies. https://blog.rust-lang.org/2023/06/01/Rust-1.70.0.html#sparse-by-default-for-cratesio So, it's better not to share the cache for "~/.cargo" -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3868) Check consistency between the doc comment in lib.rs and README.md
Kousuke Saruta created AVRO-3868: Summary: Check consistency between the doc comment in lib.rs and README.md Key: AVRO-3868 URL: https://issues.apache.org/jira/browse/AVRO-3868 Project: Apache Avro Issue Type: Improvement Components: build, rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta AVRO-3849 added cargo-rdme. So it's nice if we can check the consistency between avro/src/lib.rs and avro/README.md -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3867) Link to the Python API doc is broken
Kousuke Saruta created AVRO-3867: Summary: Link to the Python API doc is broken Key: AVRO-3867 URL: https://issues.apache.org/jira/browse/AVRO-3867 Project: Apache Avro Issue Type: Bug Reporter: Kousuke Saruta I built the website by ./build.sh dist and then tried to access to the Python API doc but the link seems broken. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3866) [Build][Python] Files are leftover after a build
Kousuke Saruta created AVRO-3866: Summary: [Build][Python] Files are leftover after a build Key: AVRO-3866 URL: https://issues.apache.org/jira/browse/AVRO-3866 Project: Apache Avro Issue Type: Bug Reporter: Kousuke Saruta Similar to AVRO-3865, lang/py/VERSION.txt is leftover after lang/py/build.sh doc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3863) Delete temporary test data after tests finish
Kousuke Saruta created AVRO-3863:
Summary: Delete temporary test data after tests finish
Key: AVRO-3863
URL: https://issues.apache.org/jira/browse/AVRO-3863
Project: Apache Avro
Issue Type: Improvement
Components: java
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
Tests for Java SDK creates some test data, which are left even after tests
finish.
{code}
ls -1 /tmp/*.avro
/tmp/junit1533190586260098046testMappedByteBuffer.avro
/tmp/junit3099644739767498712testMappedByteBuffer.avro
/tmp/junit4466003251314064556testMappedByteBuffer.avro
/tmp/junit4974226498248565286testMappedByteBuffer.avro
/tmp/junit6473921034404349045testMappedByteBuffer.avro
/tmp/junit8662732084083941415testMappedByteBuffer.avro
/tmp/random.avro
/tmp/testIgnoreSchemaValidationOnRead275054571669736256.avro
/tmp/testIgnoreSchemaValidationOnRead4615547521362396523.avro
/tmp/testIgnoreSchemaValidationOnRead4955268403025511495.avro
/tmp/testIgnoreSchemaValidationOnRead5426593551205571746.avro
/tmp/testIgnoreSchemaValidationOnRead7554021276748093417.avro
/tmp/testIgnoreSchemaValidationOnRead8241302423385070851.avro
/tmp/testInputStreamEOF3549506421974960237.avro
/tmp/testInputStreamEOF4423343183305481378.avro
/tmp/testInputStreamEOF7397178073669402143.avro
/tmp/testInputStreamEOF8065492409408481522.avro
/tmp/testInputStreamEOF8087280538995909098.avro
/tmp/testInputStreamEOF8719004614093216771.avro
/tmp/testInvalidMagicBytes1940432228654910095.avro
/tmp/testInvalidMagicBytes2703760186774533143.avro
/tmp/testInvalidMagicBytes5088097518917799234.avro
/tmp/testInvalidMagicBytes863787801374013591.avro
/tmp/testInvalidMagicBytes887543761182735490.avro
/tmp/testInvalidMagicBytes980334707534164945.avro
/tmp/testInvalidMagicLength1346115615984572207.avro
/tmp/testInvalidMagicLength1511998921770126285.avro
/tmp/testInvalidMagicLength1824057536245960603.avro
/tmp/testInvalidMagicLength2005669502062311523.avro
/tmp/testInvalidMagicLength7068591900276715585.avro
/tmp/testInvalidMagicLength8356756206873381473.avro
/tmp/testThrottledInputStream2962195154373996754.avro
/tmp/testThrottledInputStream3610702487927451328.avro
/tmp/testThrottledInputStream4661398720877824185.avro
/tmp/testThrottledInputStream5592809458916764863.avro
/tmp/testThrottledInputStream648963793454476.avro
/tmp/testThrottledInputStream8013323018361761899.avro
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3862) Add aliases and doc methods to Schema in Rust SDK
[
https://issues.apache.org/jira/browse/AVRO-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3862:
-
Priority: Minor (was: Major)
> Add aliases and doc methods to Schema in Rust SDK
> -
>
> Key: AVRO-3862
> URL: https://issues.apache.org/jira/browse/AVRO-3862
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Minor
>
> Named types (Record, Enum and Fixed) have common attributes {*}name{*},
> *aliases* and {*}doc{*}.
> We have already have *fn name* in Schema so it's nice to have *fn aliases*
> and *fn doc* too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3862) Add aliases and doc methods to Schema in Rust SDK
Kousuke Saruta created AVRO-3862:
Summary: Add aliases and doc methods to Schema in Rust SDK
Key: AVRO-3862
URL: https://issues.apache.org/jira/browse/AVRO-3862
Project: Apache Avro
Issue Type: Improvement
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
Named types (Record, Enum and Fixed) have common attributes {*}name{*},
*aliases* and {*}doc{*}.
We have already have *fn name* in Schema so it's nice to have *fn aliases* and
*fn doc* too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] Release Apache Avro 1.11.3 RC1
+1 (non-binding) Kousuke 2023年9月17日(日) 17:40 Ryan Skraba : > Hi everyone, > > I'd like to propose the following RC1 to be released as the official Apache > Avro 1.11.3 release. > > The commit id is 35ff8b997738e4d983871902d47bfb67b3250734 > * This corresponds to the tag: release-1.11.3-rc1 > * https://github.com/apache/avro/releases/tag/release-1.11.3-rc1 > > The release tarball, signature, and checksums are here (revision r64034) > * https://dist.apache.org/repos/dist/dev/avro/avro-1.11.3-rc1/ > > You can find the KEYS file here: > * https://dist.apache.org/repos/dist/dev/avro/KEYS > > Binary artifacts for Java are staged in Nexus here: > * > https://repository.apache.org/content/groups/staging/org/apache/avro/avro/1.11.3/ > > This release includes ~40 Jira issues: > * > https://issues.apache.org/jira/issues/?jql=project%3DAVRO%20AND%20fixVersion%3D1.11.3 > > Please download, verify, and test. This vote will remain open for at least > 72 hours. I would like to close at 11h59 UTC Thursday, September 21th, > 2023. > > [ ] +1 Release this as Apache Avro 1.11.3 > [ ] +0 > [ ] -1 Do not release this because... > > Best regards, > Ryan Skraba > [Countdown to vote]: > https://www.timeanddate.com/countdown/to?iso=20230921T1159=1440 >
Re: [DISCUSS] Release Avro 1.11.3
Hi Ryan, > On the other hand, I'm having some trouble getting the website built > properly. This can be independent of the release artifacts, but > you'll notice that it's still on 1.11.1! If you have ideas on how we > can simplify this process, I'd love to hear them. > I succeeded in building the website with some fixes. I've opened a PR which includes the fixes. https://github.com/apache/avro/pull/2497 Thanks, Kousuke
[jira] [Created] (AVRO-3857) Automate dist build
Kousuke Saruta created AVRO-3857: Summary: Automate dist build Key: AVRO-3857 URL: https://issues.apache.org/jira/browse/AVRO-3857 Project: Apache Avro Issue Type: Improvement Components: build Affects Versions: 1.12.0 Reporter: Kousuke Saruta Although we can build dist by ./build.sh dist, currently it's not fully automated. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3853) Suport Local timestamp logical types for the Rust SDK
Kousuke Saruta created AVRO-3853: Summary: Suport Local timestamp logical types for the Rust SDK Key: AVRO-3853 URL: https://issues.apache.org/jira/browse/AVRO-3853 Project: Apache Avro Issue Type: Improvement Components: rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta Local timestamp logical types are missing in the Rust SDK. It's nice to support them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3851) Validate default value for record fields and enums on parsing
[ https://issues.apache.org/jira/browse/AVRO-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated AVRO-3851: - Affects Version/s: 1.12.0 > Validate default value for record fields and enums on parsing > - > > Key: AVRO-3851 > URL: https://issues.apache.org/jira/browse/AVRO-3851 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.12.0 > Reporter: Kousuke Saruta >Priority: Major > > Currently, default values for record fields are not validated on parsing > except for union type fields. > Similarly, default values for enum are not also validated on parsing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3851) Validate default value for record fields and enums on parsing
Kousuke Saruta created AVRO-3851: Summary: Validate default value for record fields and enums on parsing Key: AVRO-3851 URL: https://issues.apache.org/jira/browse/AVRO-3851 Project: Apache Avro Issue Type: Improvement Components: rust Reporter: Kousuke Saruta Currently, default values for record fields are not validated on parsing except for union type fields. Similarly, default values for enum are not also validated on parsing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (AVRO-3850) Don't publish Cargo.lock
[ https://issues.apache.org/jira/browse/AVRO-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved AVRO-3850. -- Resolution: Not A Problem Close for now. See https://github.com/apache/avro/pull/2476#issuecomment-1704772007 > Don't publish Cargo.lock > > > Key: AVRO-3850 > URL: https://issues.apache.org/jira/browse/AVRO-3850 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.12.0 > Reporter: Kousuke Saruta >Priority: Minor > > Currently, Cargo.lock is published but it should not be because all the > crates are libraries. > https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3850) Don't publish Cargo.lock
Kousuke Saruta created AVRO-3850: Summary: Don't publish Cargo.lock Key: AVRO-3850 URL: https://issues.apache.org/jira/browse/AVRO-3850 Project: Apache Avro Issue Type: Improvement Components: rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta Currently, Cargo.lock is published but it should not be because all the crates are libraries. https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3849) make readme doesn't work
Kousuke Saruta created AVRO-3849: Summary: make readme doesn't work Key: AVRO-3849 URL: https://issues.apache.org/jira/browse/AVRO-3849 Project: Apache Avro Issue Type: Bug Components: doc, rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta In the Rust binding, avro/README.md is intended to be created by make readme. But it doesn't work due to some reasons. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3847) Record field doesn't accept default value if field type is union and the type of default value is pre-defined name
[
https://issues.apache.org/jira/browse/AVRO-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3847:
-
Affects Version/s: 1.12.0
> Record field doesn't accept default value if field type is union and the type
> of default value is pre-defined name
> --
>
> Key: AVRO-3847
> URL: https://issues.apache.org/jira/browse/AVRO-3847
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
>
> Given we have a schema like as follows.
> {code}
> {
> "name": "record1",
> "type": "record",
> "fields": [
> {
> "name": "f1",
> "type": {
> "name": "record2",
> "type": "record",
> "fields": [
> {
> "name": "f1_1",
> "type": "int"
> }
> ]
> }
> }, {
> "name": "f2",
> "type": ["record2", "int"],
> "default": {
> "f1_1": 100
> }
> }
> ]
> }
> {code}
> The type of the field f2 is union of record2 and int, and the default value
> is of a value of record2, which is pre-defined.
> Current Rust binding doesn't accept such schemas, raising a error message
> like as follows.
> {code}
> Error: One union type Ref must match the `default`'s value type Map
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3847) Record field doesn't accept default value if field type is union and the type of default value is pre-defined name
Kousuke Saruta created AVRO-3847:
Summary: Record field doesn't accept default value if field type
is union and the type of default value is pre-defined name
Key: AVRO-3847
URL: https://issues.apache.org/jira/browse/AVRO-3847
Project: Apache Avro
Issue Type: Bug
Components: rust
Reporter: Kousuke Saruta
Given we have a schema like as follows.
{code}
{
"name": "record1",
"type": "record",
"fields": [
{
"name": "f1",
"type": {
"name": "record2",
"type": "record",
"fields": [
{
"name": "f1_1",
"type": "int"
}
]
}
}, {
"name": "f2",
"type": ["record2", "int"],
"default": {
"f1_1": 100
}
}
]
}
{code}
The type of the field f2 is union of record2 and int, and the default value is
of a value of record2, which is pre-defined.
Current Rust binding doesn't accept such schemas, raising a error message like
as follows.
{code}
Error: One union type Ref must match the `default`'s value type Map
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3846) Race condition can happen among serde tests
[ https://issues.apache.org/jira/browse/AVRO-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated AVRO-3846: - Description: Sometimes one of tests named avro_3747* fails. You can easily reproduce this issue by cargo test avro_3747. These tests are run concurrently by Cargo test and those tests load/store the same atomic variable so This seems race condition was: Sometimes one of tests named avro_3747 fails. You can easily reproduce this issue by cargo test avro_3747. These tests are run concurrently by Cargo test and those tests load/store the same atomic variable so This seems race condition > Race condition can happen among serde tests > --- > > Key: AVRO-3846 > URL: https://issues.apache.org/jira/browse/AVRO-3846 > Project: Apache Avro > Issue Type: Bug > Components: rust >Affects Versions: 1.12.0 > Reporter: Kousuke Saruta >Priority: Major > > Sometimes one of tests named avro_3747* fails. > You can easily reproduce this issue by cargo test avro_3747. > These tests are run concurrently by Cargo test and those tests load/store the > same atomic variable so This seems race condition -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3846) Race condition can happen among serde tests
[ https://issues.apache.org/jira/browse/AVRO-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated AVRO-3846: - Description: Sometimes one of tests named avro_3747 fails. You can easily reproduce this issue by cargo test avro_3747. These tests are run concurrently by Cargo test and those tests load/store the same atomic variable so This seems race condition was: Sometimes one of tests named avro_3747 fails. These tests are run concurrently by Cargo test and those tests load/store the same atomic variable so This seems race condition > Race condition can happen among serde tests > --- > > Key: AVRO-3846 > URL: https://issues.apache.org/jira/browse/AVRO-3846 > Project: Apache Avro > Issue Type: Bug > Components: rust >Affects Versions: 1.12.0 > Reporter: Kousuke Saruta >Priority: Major > > Sometimes one of tests named avro_3747 fails. > You can easily reproduce this issue by cargo test avro_3747. > These tests are run concurrently by Cargo test and those tests load/store the > same atomic variable so This seems race condition -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3846) Race condition can happen among serde tests
Kousuke Saruta created AVRO-3846: Summary: Race condition can happen among serde tests Key: AVRO-3846 URL: https://issues.apache.org/jira/browse/AVRO-3846 Project: Apache Avro Issue Type: Bug Components: rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta Sometimes one of tests named avro_3747 fails. These tests are run concurrently by Cargo test and those tests load/store the same atomic variable so This seems race condition -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AVRO-3830) Handle namespace properly if a name starts with dot
[
https://issues.apache.org/jira/browse/AVRO-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758680#comment-17758680
]
Kousuke Saruta commented on AVRO-3830:
--
[~stestagg]
Hmm, will you fix that issue by yourself?
> Handle namespace properly if a name starts with dot
> ---
>
> Key: AVRO-3830
> URL: https://issues.apache.org/jira/browse/AVRO-3830
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The specification says about the name and namespace like as follows.
> ??The empty string may also be used as a namespace to indicate the null
> namespace??
> ??If the name specified contains a dot, then it is assumed to be a fullname,
> and any namespace also specified is ignored??
> According to this specification, if a name in a name field starts with a dot,
> it's considered that the namespace is null and the corresponding namespace
> field should be ignored.
> For example, given the following schema.
> {code}
> {
> "name": ".record1",
> "namespace": "ns1",
> "type": "record",
> "fields": []
> }
> {code}
> The name and namespace should be "record1" and null respectively.
> But the namespace is considered as "ns1" in the current Rust binding .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (AVRO-3830) Handle namespace properly if a name starts with dot
[
https://issues.apache.org/jira/browse/AVRO-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758139#comment-17758139
]
Kousuke Saruta commented on AVRO-3830:
--
[~stestagg]
{code}
The null namespace may not be used in a dot-separated sequence of names. So the
grammar for a namespace is:
| [()*]
{code}
This is about namespace, and the problem this ticket discuss is about namespace
portion of fullname.
namespace allows empty so I think it follows the specification.
> Handle namespace properly if a name starts with dot
> ---
>
> Key: AVRO-3830
> URL: https://issues.apache.org/jira/browse/AVRO-3830
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The specification says about the name and namespace like as follows.
> ??The empty string may also be used as a namespace to indicate the null
> namespace??
> ??If the name specified contains a dot, then it is assumed to be a fullname,
> and any namespace also specified is ignored??
> According to this specification, if a name in a name field starts with a dot,
> it's considered that the namespace is null and the corresponding namespace
> field should be ignored.
> For example, given the following schema.
> {code}
> {
> "name": ".record1",
> "namespace": "ns1",
> "type": "record",
> "fields": []
> }
> {code}
> The name and namespace should be "record1" and null respectively.
> But the namespace is considered as "ns1" in the current Rust binding .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3841) Align the specification of the way to encode NaN to the actual implementations
[
https://issues.apache.org/jira/browse/AVRO-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3841:
-
Summary: Align the specification of the way to encode NaN to the actual
implementations (was: Align the specification of encoding NaN to the actual
implementations)
> Align the specification of the way to encode NaN to the actual implementations
> --
>
> Key: AVRO-3841
> URL: https://issues.apache.org/jira/browse/AVRO-3841
> Project: Apache Avro
> Issue Type: Improvement
> Components: spec
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Minor
>
> The specification says about the way to encode float/double like as follows.
> {code}
> a float is written as 4 bytes. The float is converted into a 32-bit integer
> using a method equivalent to Java’s floatToIntBits and then encoded in
> little-endian format.
> a double is written as 8 bytes. The double is converted into a 64-bit integer
> using a method equivalent to Java’s doubleToLongBits and then encoded in
> little-endian format.
> {code}
> But the actual implementation in Java uses
> floatToRawIntBits/doubleToRawLongBits rather than
> floatToIntBits/doubleToLongBits.
> The they are different in the way to encode NaN.
> floatToIntBits/doubleToLongBits doesn't distinguish between NaN and -NaN but
> floatToRawIntBits/doubleToRawLongBits does.
> I confirmed all the implementation distinguish between NaN and -NaN.
> So, I think it's better to modify the specification.
> Java
> {code}
> public static int encodeFloat(float f, byte[] buf, int pos) {
> final int bits = Float.floatToRawIntBits(f);
> buf[pos + 3] = (byte) (bits >>> 24);
> buf[pos + 2] = (byte) (bits >>> 16);
> buf[pos + 1] = (byte) (bits >>> 8);
> buf[pos] = (byte) (bits);
> return 4;
> }
> public static int encodeDouble(double d, byte[] buf, int pos) {
> final long bits = Double.doubleToRawLongBits(d);
> int first = (int) (bits & 0x);
> int second = (int) ((bits >>> 32) & 0x);
> // the compiler seems to execute this order the best, likely due to
> // register allocation -- the lifetime of constants is minimized.
> buf[pos] = (byte) (first);
> buf[pos + 4] = (byte) (second);
> buf[pos + 5] = (byte) (second >>> 8);
> buf[pos + 1] = (byte) (first >>> 8);
> buf[pos + 2] = (byte) (first >>> 16);
> buf[pos + 6] = (byte) (second >>> 16);
> buf[pos + 7] = (byte) (second >>> 24);
> buf[pos + 3] = (byte) (first >>> 24);
> return 8;
> }
> {code}
> Rust
> {code}
> Value::Float(x) => buffer.extend_from_slice(_le_bytes()),
> Value::Double(x) => buffer.extend_from_slice(_le_bytes()),
> {code}
> Python
> {code}
> def write_float(self, datum: float) -> None:
>
> """
>
> A float is written as 4 bytes.
>
> The float is converted into a 32-bit integer using a method
> equivalent to
> Java's floatToIntBits and then encoded in little-endian format.
>
> """
>
> self.write(STRUCT_FLOAT.pack(datum))
> def write_double(self, datum: float) -> None:
>
> """
>
> A double is written as 8 bytes.
>
> The double is converted into a 64-bit integer using a method
> equivalent to
> Java's doubleToLongBits and then encoded in little-en
[jira] [Created] (AVRO-3841) Align the specification of encoding NaN to the actual implementations
Kousuke Saruta created AVRO-3841:
Summary: Align the specification of encoding NaN to the actual
implementations
Key: AVRO-3841
URL: https://issues.apache.org/jira/browse/AVRO-3841
Project: Apache Avro
Issue Type: Bug
Components: spec
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
The specification says about the way to encode float/double like as follows.
{code}
a float is written as 4 bytes. The float is converted into a 32-bit integer
using a method equivalent to Java’s floatToIntBits and then encoded in
little-endian format.
a double is written as 8 bytes. The double is converted into a 64-bit integer
using a method equivalent to Java’s doubleToLongBits and then encoded in
little-endian format.
{code}
But the actual implementation in Java uses
floatToRawIntBits/doubleToRawLongBits rather than
floatToIntBits/doubleToLongBits.
The they are different in the way to encode NaN.
floatToIntBits/doubleToLongBits doesn't distinguish between NaN and -NaN but
floatToRawIntBits/doubleToRawLongBits does.
I confirmed all the implementation distinguish between NaN and -NaN.
So, I think it's better to modify the specification.
Java
{code}
public static int encodeFloat(float f, byte[] buf, int pos) {
final int bits = Float.floatToRawIntBits(f);
buf[pos + 3] = (byte) (bits >>> 24);
buf[pos + 2] = (byte) (bits >>> 16);
buf[pos + 1] = (byte) (bits >>> 8);
buf[pos] = (byte) (bits);
return 4;
}
public static int encodeDouble(double d, byte[] buf, int pos) {
final long bits = Double.doubleToRawLongBits(d);
int first = (int) (bits & 0x);
int second = (int) ((bits >>> 32) & 0x);
// the compiler seems to execute this order the best, likely due to
// register allocation -- the lifetime of constants is minimized.
buf[pos] = (byte) (first);
buf[pos + 4] = (byte) (second);
buf[pos + 5] = (byte) (second >>> 8);
buf[pos + 1] = (byte) (first >>> 8);
buf[pos + 2] = (byte) (first >>> 16);
buf[pos + 6] = (byte) (second >>> 16);
buf[pos + 7] = (byte) (second >>> 24);
buf[pos + 3] = (byte) (first >>> 24);
return 8;
}
{code}
Rust
{code}
Value::Float(x) => buffer.extend_from_slice(_le_bytes()),
Value::Double(x) => buffer.extend_from_slice(_le_bytes()),
{code}
Python
{code}
def write_float(self, datum: float) -> None:
"""
A float is written as 4 bytes.
The float is converted into a 32-bit integer using a method equivalent
to
Java's floatToIntBits and then encoded in little-endian format.
"""
self.write(STRUCT_FLOAT.pack(datum))
def write_double(self, datum: float) -> None:
"""
A double is written as 8 bytes.
The double is converted into a 64-bit integer using a method equivalent
to
Java's doubleToLongBits and then encoded in little-endian format.
"""
self.write(STRUCT_DOUBLE.pack(datum))
{code}
C
{code}
static int write_float(avro_writer_t writer, const float f)
{
#if AVRO_PLATFORM_IS_BIG_ENDIAN
uint8_t buf[4];
#endif
union {
float f;
int32_t i;
} v;
v.f = f;
#if AVRO_PLATFORM_IS_BIG_ENDIAN
buf[0] = (uint8_t) (v.i >> 0);
buf[1] = (uint8_t) (v.i >> 8);
buf[2] = (uint8_t) (v.i >> 16);
buf[3] = (uint8_t) (v.i >> 24);
AVRO_WRITE(writer, buf, 4);
#else
AVRO_WR
[jira] [Updated] (AVRO-3841) Align the specification of encoding NaN to the actual implementations
[
https://issues.apache.org/jira/browse/AVRO-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3841:
-
Issue Type: Improvement (was: Bug)
> Align the specification of encoding NaN to the actual implementations
> -
>
> Key: AVRO-3841
> URL: https://issues.apache.org/jira/browse/AVRO-3841
> Project: Apache Avro
> Issue Type: Improvement
> Components: spec
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Minor
>
> The specification says about the way to encode float/double like as follows.
> {code}
> a float is written as 4 bytes. The float is converted into a 32-bit integer
> using a method equivalent to Java’s floatToIntBits and then encoded in
> little-endian format.
> a double is written as 8 bytes. The double is converted into a 64-bit integer
> using a method equivalent to Java’s doubleToLongBits and then encoded in
> little-endian format.
> {code}
> But the actual implementation in Java uses
> floatToRawIntBits/doubleToRawLongBits rather than
> floatToIntBits/doubleToLongBits.
> The they are different in the way to encode NaN.
> floatToIntBits/doubleToLongBits doesn't distinguish between NaN and -NaN but
> floatToRawIntBits/doubleToRawLongBits does.
> I confirmed all the implementation distinguish between NaN and -NaN.
> So, I think it's better to modify the specification.
> Java
> {code}
> public static int encodeFloat(float f, byte[] buf, int pos) {
> final int bits = Float.floatToRawIntBits(f);
> buf[pos + 3] = (byte) (bits >>> 24);
> buf[pos + 2] = (byte) (bits >>> 16);
> buf[pos + 1] = (byte) (bits >>> 8);
> buf[pos] = (byte) (bits);
> return 4;
> }
> public static int encodeDouble(double d, byte[] buf, int pos) {
> final long bits = Double.doubleToRawLongBits(d);
> int first = (int) (bits & 0x);
> int second = (int) ((bits >>> 32) & 0x);
> // the compiler seems to execute this order the best, likely due to
> // register allocation -- the lifetime of constants is minimized.
> buf[pos] = (byte) (first);
> buf[pos + 4] = (byte) (second);
> buf[pos + 5] = (byte) (second >>> 8);
> buf[pos + 1] = (byte) (first >>> 8);
> buf[pos + 2] = (byte) (first >>> 16);
> buf[pos + 6] = (byte) (second >>> 16);
> buf[pos + 7] = (byte) (second >>> 24);
> buf[pos + 3] = (byte) (first >>> 24);
> return 8;
> }
> {code}
> Rust
> {code}
> Value::Float(x) => buffer.extend_from_slice(_le_bytes()),
> Value::Double(x) => buffer.extend_from_slice(_le_bytes()),
> {code}
> Python
> {code}
> def write_float(self, datum: float) -> None:
>
> """
>
> A float is written as 4 bytes.
>
> The float is converted into a 32-bit integer using a method
> equivalent to
> Java's floatToIntBits and then encoded in little-endian format.
>
> """
>
> self.write(STRUCT_FLOAT.pack(datum))
> def write_double(self, datum: float) -> None:
>
> """
>
> A double is written as 8 bytes.
>
> The double is converted into a 64-bit integer using a method
> equivalent to
> Java's doubleToLongBits and then encoded in little-endian format.
>
> """
[jira] [Updated] (AVRO-3837) Disallow invalid namespaces for the Rust binding
[
https://issues.apache.org/jira/browse/AVRO-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3837:
-
Description:
The current Rust binding doesn't accept invalid namespaces if such namespaces
are in a name field.
{code}
{
"name": "ns1.invalid-ns.record1",
"type": "record"
"fields": []
}
{code}
But, even if a invalid namespace is in a namespace field, the Rust binding
accept such namespaces.
{code}
{
"name": "record1",
"namespace": "ns1.invalid-ns",
"type": "record",
"fields": []
}
{code}
was:
The current Rust binding doesn't accept invalid namespaces if such namespaces
are in a name field.
{code}
{
"name": "ns1.invalid-ns.record1",
"type": "record"
"fields": []
}
{code}
But, even if a invalid namespace is in a namespace field, the Rust binding
accept such namespaces.
{code}
"name": "record1",
"namespace": "ns1.invalid-ns",
"type": "record",
"fields": []
}
{code}
> Disallow invalid namespaces for the Rust binding
>
>
> Key: AVRO-3837
> URL: https://issues.apache.org/jira/browse/AVRO-3837
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.12.0
>Reporter: Kousuke Saruta
>Priority: Major
>
> The current Rust binding doesn't accept invalid namespaces if such namespaces
> are in a name field.
> {code}
> {
> "name": "ns1.invalid-ns.record1",
> "type": "record"
> "fields": []
> }
> {code}
> But, even if a invalid namespace is in a namespace field, the Rust binding
> accept such namespaces.
> {code}
> {
> "name": "record1",
> "namespace": "ns1.invalid-ns",
> "type": "record",
> "fields": []
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3837) Disallow invalid namespaces for the Rust binding
[
https://issues.apache.org/jira/browse/AVRO-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3837:
-
Description:
The current Rust binding doesn't accept invalid namespaces if such namespaces
are in a name field.
{code}
{
"name": "ns1.invalid-ns.record1",
"type": "record"
"fields": []
}
{code}
But, even if a invalid namespace is in a namespace field, the Rust binding
accept such namespaces.
{code}
"name": "record1",
"namespace": "ns1.invalid-ns",
"type": "record",
"fields": []
}
{code}
was:
The current Rust binding doesn't accept invalid namespaces if such namespaces
are in name field.
{code}
{
"name": "ns1.invalid-ns.record1",
"type": "record"
"fields": []
}
{code}
But if a invalid namespace in namespace field doesn't validate.
{code}
"name": "record1",
"namespace": "ns1.invalid-ns",
"type": "record",
"fields": []
}
{code}
> Disallow invalid namespaces for the Rust binding
>
>
> Key: AVRO-3837
> URL: https://issues.apache.org/jira/browse/AVRO-3837
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.12.0
>Reporter: Kousuke Saruta
>Priority: Major
>
> The current Rust binding doesn't accept invalid namespaces if such namespaces
> are in a name field.
> {code}
> {
> "name": "ns1.invalid-ns.record1",
> "type": "record"
> "fields": []
> }
> {code}
> But, even if a invalid namespace is in a namespace field, the Rust binding
> accept such namespaces.
> {code}
> "name": "record1",
> "namespace": "ns1.invalid-ns",
> "type": "record",
> "fields": []
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3837) Disallow invalid namespaces for the Rust binding
[
https://issues.apache.org/jira/browse/AVRO-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3837:
-
Summary: Disallow invalid namespaces for the Rust binding (was: Disallow
invalid namespace for the Rust binding)
> Disallow invalid namespaces for the Rust binding
>
>
> Key: AVRO-3837
> URL: https://issues.apache.org/jira/browse/AVRO-3837
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
>
> The current Rust binding doesn't accept invalid namespaces if such namespaces
> are in name field.
> {code}
> {
> "name": "ns1.invalid-ns.record1",
> "type": "record"
> "fields": []
> }
> {code}
> But if a invalid namespace in namespace field doesn't validate.
> {code}
> "name": "record1",
> "namespace": "ns1.invalid-ns",
> "type": "record",
> "fields": []
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3837) Disallow invalid namespace for the Rust binding
Kousuke Saruta created AVRO-3837:
Summary: Disallow invalid namespace for the Rust binding
Key: AVRO-3837
URL: https://issues.apache.org/jira/browse/AVRO-3837
Project: Apache Avro
Issue Type: Bug
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
The current Rust binding doesn't accept invalid namespaces if such namespaces
are in name field.
{code}
{
"name": "ns1.invalid-ns.record1",
"type": "record"
"fields": []
}
{code}
But if a invalid namespace in namespace field doesn't validate.
{code}
"name": "record1",
"namespace": "ns1.invalid-ns",
"type": "record",
"fields": []
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Specification of namespaces
Let me correct.
Not "alias", but "aliases".
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","
>> type":{"type":"enum","name":"exampleEnum","alias":"1
>
> bad alias","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1
> bad alias"}]}
>
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","
type":{"type":"enum","name":"exampleEnum","aliases":["1
bad alias"],"symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1
bad alias"}]}
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1bad
> alias.foo.bar","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1bad
> alias.foo.bar"}]}
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","aliases":["1bad
alias.foo.bar"],"symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1bad
alias.foo.bar"}]}
2023年8月18日(金) 15:37 Kousuke Saruta :
> Hi Michael,
>
>
>> {"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1
>> bad alias","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1
>> bad alias"}]}
>>
>
> In the current Java binding, the namespace portion in an alias is accepted
> without validation.
> So, the following schema is acceptable.
>
> {"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1bad
> alias.foo.bar","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"exampleEnum"}]}
>
> I'm discussing namespace in this thread, so this behavior seems O.K to me.
>
> But reference to another named types is not implemented for the Java
> binding.
> So the following schema is not accepted.
>
> {"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1bad
> alias.foo.bar","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1bad
> alias.foo.bar"}]}
>
> I have a plan to fix it.
>
> 2023年8月18日(金) 11:33 Michael A. Smith :
>
>> I found I'm still a little confused at how using aliases to correct
>> invalid names should work. Maybe you can define an alias that is an
>> invalid name, but having done so, can you use it? I tried this schema
>> in both the Python and Java implementations.
>>
>>
>> {"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1
>> bad alias","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1
>> bad alias"}]}
>>
>> I expected it to error in Python, because I know Python requires valid
>> names for aliases. But Java also errored with "schema failed: Illegal
>> initial character: 1 bad alias". I am not sure if the error is from
>> the alias definition or its use.
>>
>> If my example is flawed, can someone supply a correct one?
>>
>> On Thu, Aug 17, 2023 at 4:53 AM Oscar Westra van Holthe - Kind
>> wrote:
>> >
>> > On Mon, 14 Aug 2023 at 14:11, Ryan Skraba wrote:
>> >
>> > > I think the right thing to do is [to] use a system
>> > > property / schema aliases to help people migrate back to the correct
>> > > behaviour. If you are actually using Avro/Protobuf together, you
>> > > might be the best person to help us figure out the right was to do
>> > > this migration!
>> > >
>> >
>> > The idea that aliases can be used to evolve a schema with invalid names
>> to
>> > a schema with valid names is a sensible one, and currently hidden in the
>> > schema resolution rules in the specification.
>> >
>> > I've added AVRO-3833 <https://issues.apache.org/jira/browse/AVRO-3833>
>> (with
>> > PR <https://github.com/apache/avro/pull/2448>) because I wanted to
>> clarify
>> > that names must
>> > be unique (because otherwise schema resolution cannot work), and that
>> this
>> > includes aliases. The change also includes this migration/fix option.
>> >
>> > Kind regards,
>> > Oscar
>> >
>> > --
>> > ✉️ Oscar Westra van Holthe - Kind
>>
>
Re: Specification of namespaces
Hi Michael,
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1
> bad alias","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1
> bad alias"}]}
>
In the current Java binding, the namespace portion in an alias is accepted
without validation.
So, the following schema is acceptable.
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1bad
alias.foo.bar","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"exampleEnum"}]}
I'm discussing namespace in this thread, so this behavior seems O.K to me.
But reference to another named types is not implemented for the Java
binding.
So the following schema is not accepted.
{"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1bad
alias.foo.bar","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1bad
alias.foo.bar"}]}
I have a plan to fix it.
2023年8月18日(金) 11:33 Michael A. Smith :
> I found I'm still a little confused at how using aliases to correct
> invalid names should work. Maybe you can define an alias that is an
> invalid name, but having done so, can you use it? I tried this schema
> in both the Python and Java implementations.
>
>
> {"type":"record","name":"AliasReferenceExample","fields":[{"name":"anEnum","type":{"type":"enum","name":"exampleEnum","alias":"1
> bad alias","symbols":["A","B","C"]}},{"name":"anotherEnum","type":"1
> bad alias"}]}
>
> I expected it to error in Python, because I know Python requires valid
> names for aliases. But Java also errored with "schema failed: Illegal
> initial character: 1 bad alias". I am not sure if the error is from
> the alias definition or its use.
>
> If my example is flawed, can someone supply a correct one?
>
> On Thu, Aug 17, 2023 at 4:53 AM Oscar Westra van Holthe - Kind
> wrote:
> >
> > On Mon, 14 Aug 2023 at 14:11, Ryan Skraba wrote:
> >
> > > I think the right thing to do is [to] use a system
> > > property / schema aliases to help people migrate back to the correct
> > > behaviour. If you are actually using Avro/Protobuf together, you
> > > might be the best person to help us figure out the right was to do
> > > this migration!
> > >
> >
> > The idea that aliases can be used to evolve a schema with invalid names
> to
> > a schema with valid names is a sensible one, and currently hidden in the
> > schema resolution rules in the specification.
> >
> > I've added AVRO-3833
> (with
> > PR ) because I wanted to
> clarify
> > that names must
> > be unique (because otherwise schema resolution cannot work), and that
> this
> > includes aliases. The change also includes this migration/fix option.
> >
> > Kind regards,
> > Oscar
> >
> > --
> > ✉️ Oscar Westra van Holthe - Kind
>
[jira] [Created] (AVRO-3832) Python test doesn't work with Docker
Kousuke Saruta created AVRO-3832: Summary: Python test doesn't work with Docker Key: AVRO-3832 URL: https://issues.apache.org/jira/browse/AVRO-3832 Project: Apache Avro Issue Type: Bug Components: python Affects Versions: 1.12.0 Reporter: Kousuke Saruta Python test doesn't work successfully with ./build.sh docker-test. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3830) Handle namespace properly if a name starts with dot
[
https://issues.apache.org/jira/browse/AVRO-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3830:
-
Summary: Handle namespace properly if a name starts with dot (was: Handle
namespace property if a name starts with dot)
> Handle namespace properly if a name starts with dot
> ---
>
> Key: AVRO-3830
> URL: https://issues.apache.org/jira/browse/AVRO-3830
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
>
> The specification says about the name and namespace like as follows.
> ??The empty string may also be used as a namespace to indicate the null
> namespace??
> ??If the name specified contains a dot, then it is assumed to be a fullname,
> and any namespace also specified is ignored??
> According to this specification, if a name in a name field starts with a dot,
> it's considered that the namespace is null and the corresponding namespace
> field should be ignored.
> For example, given the following schema.
> {code}
> {
> "name": ".record1",
> "namespace": "ns1",
> "type": "record",
> "fields": []
> }
> {code}
> The name and namespace should be "record1" and null respectively.
> But the namespace is considered as "ns1" in the current Rust binding .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3830) Handle namespace property if a name starts with dot
[
https://issues.apache.org/jira/browse/AVRO-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3830:
-
Affects Version/s: 1.12.0
> Handle namespace property if a name starts with dot
> ---
>
> Key: AVRO-3830
> URL: https://issues.apache.org/jira/browse/AVRO-3830
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
>
> The specification says about the name and namespace like as follows.
> ??The empty string may also be used as a namespace to indicate the null
> namespace??
> ??If the name specified contains a dot, then it is assumed to be a fullname,
> and any namespace also specified is ignored??
> According to this specification, if a name in a name field starts with a dot,
> it's considered that the namespace is null and the corresponding namespace
> field should be ignored.
> For example, given the following schema.
> {code}
> {
> "name": ".record1",
> "namespace": "ns1",
> "type": "record",
> "fields": []
> }
> {code}
> The name and namespace should be "record1" and null respectively.
> But the namespace is considered as "ns1" in the current Rust binding .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3830) Handle namespace property if a name starts with dot
Kousuke Saruta created AVRO-3830:
Summary: Handle namespace property if a name starts with dot
Key: AVRO-3830
URL: https://issues.apache.org/jira/browse/AVRO-3830
Project: Apache Avro
Issue Type: Bug
Reporter: Kousuke Saruta
The specification says about the name and namespace like as follows.
??The empty string may also be used as a namespace to indicate the null
namespace??
??If the name specified contains a dot, then it is assumed to be a fullname,
and any namespace also specified is ignored??
According to this specification, if a name in a name field starts with a dot,
it's considered that the namespace is null and the corresponding namespace
field should be ignored.
For example, given the following schema.
{code}
{
"name": ".record1",
"namespace": "ns1",
"type": "record",
"fields": []
}
{code}
The name and namespace should be "record1" and null respectively.
But the namespace is considered as "ns1" in the current Rust binding .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Specification of namespaces
Hi Martin,
Thank you for the comment.
> Hi,
>
> On Wed, Aug 9, 2023 at 6:30 PM Kousuke Saruta wrote:
>
> > Hi developers,
> >
> > I'd like to discuss the specification of namespace.
> > According to the specification, each dot separated portion of a namespace
> > should be [a-zA-Z_]][a-zA-Z0-9_]*.
> > https://avro.apache.org/docs/1.11.1/specification/#names
> >
> > But the actual implementations of some language bindings don't follow the
> > specification, and accept any characters.
> > Especially, the Java binding generates namespaces which contain "$" for
> > inner classes generated by protobuf.
> >
> > So, should we need to review the namespace specification?
> >
>
> To the developers who are familiar with the Java SDK: What problems do you
> see if the generator stops producing "$", i.e. do something like
> generated.replace('$', '') ?
> Would that break existing apps ?
>
If we replace "$" with any other character in the new version of Avro,
data serialized by an old Avro cannot be converted back to protobuf format,
right?
2023年8月10日(木) 16:38 Martin Grigorov :
> Hi,
>
> On Wed, Aug 9, 2023 at 6:30 PM Kousuke Saruta wrote:
>
> > Hi developers,
> >
> > I'd like to discuss the specification of namespace.
> > According to the specification, each dot separated portion of a namespace
> > should be [a-zA-Z_]][a-zA-Z0-9_]*.
> > https://avro.apache.org/docs/1.11.1/specification/#names
> >
> > But the actual implementations of some language bindings don't follow the
> > specification, and accept any characters.
> > Especially, the Java binding generates namespaces which contain "$" for
> > inner classes generated by protobuf.
> >
> > So, should we need to review the namespace specification?
> >
>
> To the developers who are familiar with the Java SDK: What problems do you
> see if the generator stops producing "$", i.e. do something like
> generated.replace('$', '') ?
> Would that break existing apps ?
>
>
>
> >
> > Thanks,
> > Kousuke
> >
>
[jira] [Updated] (AVRO-3827) Disallow duplicate field names
[
https://issues.apache.org/jira/browse/AVRO-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3827:
-
Description:
If a schema contains a record and some of its fields have the same field name,
such schema should not be allowed.
{code:java}
{
"name": "my_schema",
"type": "record",
"fields": [
{
"name": "f1",
"type": {
"name": "a",
"type": "record",
"fields": []
}
}, {
"name": "f1",
"type": {
"name": "b",
"type": "record",
"fields": []
}
}
]
}
{code}
But the current Rust binding accept.
was:
If a schema contains a record and some of its fields have the same field name,
such schema should not be allowed.
{code}
{
"name": "my_schema",
"type": "record",
"fields": [
{
"name": "f1",
"type": {
"name": "a",
"type": "record",
"fields": []
}
} {
"name": "f1",
"type": {
"name": "b",
"type": "record",
"fields": []
}
}
]
}
{code}
But the current Rust binding accept.
> Disallow duplicate field names
> --
>
> Key: AVRO-3827
> URL: https://issues.apache.org/jira/browse/AVRO-3827
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.12.0
>Reporter: Kousuke Saruta
>Priority: Major
>
> If a schema contains a record and some of its fields have the same field
> name, such schema should not be allowed.
> {code:java}
> {
> "name": "my_schema",
> "type": "record",
> "fields": [
> {
> "name": "f1",
> "type": {
> "name": "a",
> "type": "record",
> "fields": []
> }
> }, {
> "name": "f1",
> "type": {
> "name": "b",
> "type": "record",
> "fields": []
> }
> }
> ]
> }
> {code}
> But the current Rust binding accept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3827) Disallow duplicate field names
Kousuke Saruta created AVRO-3827:
Summary: Disallow duplicate field names
Key: AVRO-3827
URL: https://issues.apache.org/jira/browse/AVRO-3827
Project: Apache Avro
Issue Type: Bug
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
If a schema contains a record and some of its fields have the same field name,
such schema should not be allowed.
{code}
{
"name": "my_schema",
"type": "record",
"fields": [
{
"name": "f1",
"type": {
"name": "a",
"type": "record",
"fields": []
}
} {
"name": "f1",
"type": {
"name": "b",
"type": "record",
"fields": []
}
}
]
}
{code}
But the current Rust binding accept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Specification of namespaces
Hi developers, I'd like to discuss the specification of namespace. According to the specification, each dot separated portion of a namespace should be [a-zA-Z_]][a-zA-Z0-9_]*. https://avro.apache.org/docs/1.11.1/specification/#names But the actual implementations of some language bindings don't follow the specification, and accept any characters. Especially, the Java binding generates namespaces which contain "$" for inner classes generated by protobuf. So, should we need to review the namespace specification? Thanks, Kousuke
[jira] [Updated] (AVRO-3825) Disallow invalid namespaces
[
https://issues.apache.org/jira/browse/AVRO-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3825:
-
Description:
According to the specification, each portion of a namespace separated by dot
should be [a-zA-Z_][a-zA-Z0-9_].
[https://avro.apache.org/docs/1.11.1/specification/#names]
{code:java}
The name portion of the fullname of named types, record field names, and enum
symbols must:
start with [A-Za-z_]
subsequently contain only [A-Za-z0-9_]
A namespace is a dot-separated sequence of such names. The empty string may
also be used as a namespace to indicate the null namespace. Equality of names
(including field names and enum symbols) as well as fullnames is case-sensitive.
The null namespace may not be used in a dot-separated sequence of names. So the
grammar for a namespace is:
| [()*]
{code}
was:
According to the specification, each portion of a namespace separated by dot
should be [a-z,A-Z,_][a-z,A-Z,0-9_].
[https://avro.apache.org/docs/1.11.1/specification/#names]
{code:java}
The name portion of the fullname of named types, record field names, and enum
symbols must:
start with [A-Za-z_]
subsequently contain only [A-Za-z0-9_]
A namespace is a dot-separated sequence of such names. The empty string may
also be used as a namespace to indicate the null namespace. Equality of names
(including field names and enum symbols) as well as fullnames is case-sensitive.
The null namespace may not be used in a dot-separated sequence of names. So the
grammar for a namespace is:
| [()*]
{code}
> Disallow invalid namespaces
> ---
>
> Key: AVRO-3825
> URL: https://issues.apache.org/jira/browse/AVRO-3825
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> According to the specification, each portion of a namespace separated by dot
> should be [a-zA-Z_][a-zA-Z0-9_].
> [https://avro.apache.org/docs/1.11.1/specification/#names]
> {code:java}
> The name portion of the fullname of named types, record field names, and enum
> symbols must:
> start with [A-Za-z_]
> subsequently contain only [A-Za-z0-9_]
> A namespace is a dot-separated sequence of such names. The empty string may
> also be used as a namespace to indicate the null namespace. Equality of names
> (including field names and enum symbols) as well as fullnames is
> case-sensitive.
> The null namespace may not be used in a dot-separated sequence of names. So
> the grammar for a namespace is:
>| [()*]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3825) Disallow invalid namespace
Kousuke Saruta created AVRO-3825:
Summary: Disallow invalid namespace
Key: AVRO-3825
URL: https://issues.apache.org/jira/browse/AVRO-3825
Project: Apache Avro
Issue Type: Bug
Components: java
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
According to the specification, each portion of a namespace separated by dot
should be [a-z,A-Z,_][a-z,A-Z,0-9_].
[https://avro.apache.org/docs/1.11.1/specification/#names]
{code:java}
The name portion of the fullname of named types, record field names, and enum
symbols must:
start with [A-Za-z_]
subsequently contain only [A-Za-z0-9_]
A namespace is a dot-separated sequence of such names. The empty string may
also be used as a namespace to indicate the null namespace. Equality of names
(including field names and enum symbols) as well as fullnames is case-sensitive.
The null namespace may not be used in a dot-separated sequence of names. So the
grammar for a namespace is:
| [()*]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (AVRO-3825) Disallow invalid namespaces
[
https://issues.apache.org/jira/browse/AVRO-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3825:
-
Summary: Disallow invalid namespaces (was: Disallow invalid namespace)
> Disallow invalid namespaces
> ---
>
> Key: AVRO-3825
> URL: https://issues.apache.org/jira/browse/AVRO-3825
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
>
> According to the specification, each portion of a namespace separated by dot
> should be [a-z,A-Z,_][a-z,A-Z,0-9_].
> [https://avro.apache.org/docs/1.11.1/specification/#names]
> {code:java}
> The name portion of the fullname of named types, record field names, and enum
> symbols must:
> start with [A-Za-z_]
> subsequently contain only [A-Za-z0-9_]
> A namespace is a dot-separated sequence of such names. The empty string may
> also be used as a namespace to indicate the null namespace. Equality of names
> (including field names and enum symbols) as well as fullnames is
> case-sensitive.
> The null namespace may not be used in a dot-separated sequence of names. So
> the grammar for a namespace is:
>| [()*]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3824) The instruction for building the website should be more precise
Kousuke Saruta created AVRO-3824: Summary: The instruction for building the website should be more precise Key: AVRO-3824 URL: https://issues.apache.org/jira/browse/AVRO-3824 Project: Apache Avro Issue Type: Bug Components: doc Affects Versions: 1.12.0 Reporter: Kousuke Saruta How to build the website is instructed in doc/README.md but we cannot build even though we follow it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (AVRO-3823) Show helpful error messages
[
https://issues.apache.org/jira/browse/AVRO-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751208#comment-17751208
]
Kousuke Saruta commented on AVRO-3823:
--
[~mgrigorov] Oh, I see. I didn't know anyhow works with thiserror well (and I
noticed both are created by the same author).
> Show helpful error messages
> ---
>
> Key: AVRO-3823
> URL: https://issues.apache.org/jira/browse/AVRO-3823
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The current Rust binding doesn't show helpful error messages.
> Actually, error types are implemented with helpful error messages.
> This is an example.
> {code:java}
> #[error("No `name` field")]
> GetNameField,
> {code}
> But those error messages are not shown.
> Given we try to a invalid schema which contains no name field, we expect to
> get "No `name` field" but the actual is "GetNameFIeld", which makes it
> difficult for users to resolve the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3823) Show helpful error messages
Kousuke Saruta created AVRO-3823:
Summary: Show helpful error messages
Key: AVRO-3823
URL: https://issues.apache.org/jira/browse/AVRO-3823
Project: Apache Avro
Issue Type: Bug
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
The current Rust binding doesn't show helpful error messages.
Actually, error types are implemented with helpful error messages.
This is an example.
{code:java}
#[error("No `name` field")]
GetNameField,
{code}
But those error messages are not shown.
Given we try to a invalid schema which contains no name field, we expect to get
"No `name` field" but the actual is "GetNameFIeld", which makes it difficult
for users to resolve the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3820) Don't allow invalid field names
Kousuke Saruta created AVRO-3820:
Summary: Don't allow invalid field names
Key: AVRO-3820
URL: https://issues.apache.org/jira/browse/AVRO-3820
Project: Apache Avro
Issue Type: Bug
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
Given we have a schema where a field name doesn't match [A-Za-z_][A-Za-z0-9_]*
like f1.x.
{code}
{
"name": "my_record",
"type": "record",
"fields": [
{
"name": "f1.x",
"type": {
"name": "my_enum",
"type": "enum",
"symbols": ["a"]
}
}
]
}
{code}
The current Rust binding accepts such a schema but it seems illegal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (SPARK-44490) Remove TaskPagedTable in StagePage
[ https://issues.apache.org/jira/browse/SPARK-44490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-44490. Fix Version/s: 4.0.0 Assignee: dzcxzl Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/42085 > Remove TaskPagedTable in StagePage > -- > > Key: SPARK-44490 > URL: https://issues.apache.org/jira/browse/SPARK-44490 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 3.4.1 >Reporter: dzcxzl >Assignee: dzcxzl >Priority: Minor > Fix For: 4.0.0 > > > In [SPARK-21809|https://issues.apache.org/jira/browse/SPARK-21809], we > introduced stagespage-template.html to show the running status of Stage. > TaskPagedTable is no longer effective, but there are still many PRs updating > related codes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (AVRO-3818) Enclosing namespace should be inherited to the inner named types if they have no their own namespaces
Kousuke Saruta created AVRO-3818:
Summary: Enclosing namespace should be inherited to the inner
named types if they have no their own namespaces
Key: AVRO-3818
URL: https://issues.apache.org/jira/browse/AVRO-3818
Project: Apache Avro
Issue Type: Bug
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
Given we have a schema like as follows.
{code}
{
"namespace": "my_ns",
"type": "record",
"aliases": ["my_alias"],
"name": "my_schema",
"fields": [
{
"name": "f1",
"type": {
"name": "enum1",
"type": "enum",
"symbols": ["a"]
}
}
]
}
{code}
The canonical form of this schema should be
{code}
{"name":"my_ns.my_schema","type":"record","fields":[{"name":"f1","type":{"name":"my_ns.enum1","type":"enum","symbols":["a"]}}]}
{code}
But with the current Rust binding, enum1 doesn't inherit the enclosing
namespace "my_ns".
{code}
{"name":"my_ns.my_schema","type":"record","fields":[{"name":"f1","type":{"name":"enum1","type":"enum","symbols":["a"]}}]}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3815) Broken indentation in the specification doc
Kousuke Saruta created AVRO-3815: Summary: Broken indentation in the specification doc Key: AVRO-3815 URL: https://issues.apache.org/jira/browse/AVRO-3815 Project: Apache Avro Issue Type: Improvement Components: doc Affects Versions: 1.12.0 Reporter: Kousuke Saruta Attachments: complex-types-layout.png See the attachment. The items _order_ and _aliases_ below the table should be one more deeper indentation level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (AVRO-3812) Handle null namespace properly for canonicalized schema representation
[
https://issues.apache.org/jira/browse/AVRO-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated AVRO-3812:
-
Summary: Handle null namespace properly for canonicalized schema
representation (was: Handle null namespace properly)
> Handle null namespace properly for canonicalized schema representation
> --
>
> Key: AVRO-3812
> URL: https://issues.apache.org/jira/browse/AVRO-3812
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
>Affects Versions: 1.12.0
> Reporter: Kousuke Saruta
>Priority: Major
>
> Considering the following schema, which contains namespaces of "".
> {code}
> {
> "namespace": "",
> "type": "record",
> "name": "my_schema",
> "fields": [
>{
> "name": "a",
> "type": {
>"type": "enum",
>"name": "my_enum",
>"namespace": "",
>"symbols": ["a", "b"]
> }
>}, {
> "name": "b",
> "type": {
>"type": "fixed",
>"name": "my_fixed",
>"namespace": "",
>"size": 10
> }
>}
> ]
> }
> {code}
> If we try to canonicalize this schema with the following code
> {code}
> let schema = Schema::parse_str(schema_str).unwrap().canonical_form();
> println!("{schema}");
> {code}
> We get the following result.
> {code}
> {"name":".my_schema","type":"record","fields":[{"name":"a","type":{"name":".my_enum","type":"enum","symbols":["a","b"]}},{"name":"b","type":{"name":".my_fixed","type":"fixed","size":10}}]}
> {code}
> But .my_schema, .my_enum and .my_fixed should not starts with a dot.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3812) Handle null namespace properly
Kousuke Saruta created AVRO-3812:
Summary: Handle null namespace properly
Key: AVRO-3812
URL: https://issues.apache.org/jira/browse/AVRO-3812
Project: Apache Avro
Issue Type: Improvement
Components: rust
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
Considering the following schema, which contains namespaces of "".
{code}
{
"namespace": "",
"type": "record",
"name": "my_schema",
"fields": [
{
"name": "a",
"type": {
"type": "enum",
"name": "my_enum",
"namespace": "",
"symbols": ["a", "b"]
}
}, {
"name": "b",
"type": {
"type": "fixed",
"name": "my_fixed",
"namespace": "",
"size": 10
}
}
]
}
{code}
If we try to canonicalize this schema with the following code
{code}
let schema = Schema::parse_str(schema_str).unwrap().canonical_form();
println!("{schema}");
{code}
We get the following result.
{code}
{"name":".my_schema","type":"record","fields":[{"name":"a","type":{"name":".my_enum","type":"enum","symbols":["a","b"]}},{"name":"b","type":{"name":".my_fixed","type":"fixed","size":10}}]}
{code}
But .my_schema, .my_enum and .my_fixed should not starts with a dot.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3811) Mention about xz and zstandard as default registerd codecs in JavaDoc
Kousuke Saruta created AVRO-3811:
Summary: Mention about xz and zstandard as default registerd
codecs in JavaDoc
Key: AVRO-3811
URL: https://issues.apache.org/jira/browse/AVRO-3811
Project: Apache Avro
Issue Type: Improvement
Components: doc, java
Affects Versions: 1.12.0
Reporter: Kousuke Saruta
The JavaDoc for CodecFactory says
{code}
* Currently there are three codecs registered by default:
*
* {@code null}
* {@code deflate}
* {@code snappy}
* {@code bzip2}
*
{code}
But xz and zstandard are also registered by default.
{code}
static {
addCodec(DataFileConstants.NULL_CODEC, nullCodec());
addCodec(DataFileConstants.DEFLATE_CODEC,
deflateCodec(DEFAULT_DEFLATE_LEVEL));
addCodec(DataFileConstants.BZIP2_CODEC, bzip2Codec());
addCodec(DataFileConstants.XZ_CODEC, xzCodec(DEFAULT_XZ_LEVEL));
addCodec(DataFileConstants.ZSTANDARD_CODEC,
zstandardCodec(DEFAULT_ZSTANDARD_LEVEL, DEFAULT_ZSTANDARD_BUFFERPOOL));
addCodec(DataFileConstants.SNAPPY_CODEC, snappyCodec());
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (AVRO-3800) profile section should be declared in the root package.
Kousuke Saruta created AVRO-3800: Summary: profile section should be declared in the root package. Key: AVRO-3800 URL: https://issues.apache.org/jira/browse/AVRO-3800 Project: Apache Avro Issue Type: Bug Components: build, rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta In wasm-demo/Cargo.toml, [profile.release] is declared but it's ignored. It should be declared in Cargo.toml of root package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (AVRO-3799) Enable the schema parser to read and parse from input streams for Rust binding
Kousuke Saruta created AVRO-3799: Summary: Enable the schema parser to read and parse from input streams for Rust binding Key: AVRO-3799 URL: https://issues.apache.org/jira/browse/AVRO-3799 Project: Apache Avro Issue Type: Improvement Components: rust Affects Versions: 1.12.0 Reporter: Kousuke Saruta The Java binding supports reading and parsing schema from input streams. So, it's great if the feature is supported for the Rust binding too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (SPARK-44279) Upgrade optionator to ^0.9.3
[ https://issues.apache.org/jira/browse/SPARK-44279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-44279. Target Version/s: 3.5.0 Assignee: Bjørn Jørgensen Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/41955 > Upgrade optionator to ^0.9.3 > > > Key: SPARK-44279 > URL: https://issues.apache.org/jira/browse/SPARK-44279 > Project: Spark > Issue Type: Dependency upgrade > Components: Build >Affects Versions: 3.4.1, 3.5.0 >Reporter: Bjørn Jørgensen >Assignee: Bjørn Jørgensen >Priority: Minor > > [Regular Expression Denial of Service (ReDoS) - > CVE-2023-26115|https://github.com/jonschlinkert/word-wrap/issues/32] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44279) Upgrade optionator to ^0.9.3
[ https://issues.apache.org/jira/browse/SPARK-44279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-44279: --- Priority: Minor (was: Major) > Upgrade optionator to ^0.9.3 > > > Key: SPARK-44279 > URL: https://issues.apache.org/jira/browse/SPARK-44279 > Project: Spark > Issue Type: Dependency upgrade > Components: Build >Affects Versions: 3.4.1, 3.5.0 >Reporter: Bjørn Jørgensen >Priority: Minor > > [Regular Expression Denial of Service (ReDoS) - > CVE-2023-26115|https://github.com/jonschlinkert/word-wrap/issues/32] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[apache/incubator-teaclave] Fix deploy instruction for Azure to comply with SGX 2.17. (PR #687)
## Description The instruction for deployment on Azure doesnt work due to SGX 2.14 is used in the docker image `teaclave/teaclave-build-ubuntu-1804-sgx-2.14`. Now that SGX 2.17 is required, `teaclave/teaclave-build-ubuntu-1804-sgx-2.17.1` should be used. ## Type of change (select or add applied and delete the others) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] API change with a documentation update - [ ] Additional test coverage - [ ] Code cleanup or just sync with upstream third-party crates ## How has this been tested? Confirmed `docker run` passes with the new instruction. ## Checklist - [x] Fork the repo and create your branch from `master`. - [x] If youve added code that should be tested, add tests. - [x] If youve changed APIs, update the documentation. - [x] Ensure the tests pass (see CI results). - [x] Make sure your code lints/format. You can view, comment on, or merge this pull request online at: https://github.com/apache/incubator-teaclave/pull/687 -- Commit Summary -- * Fix deploy instruction for Azure to comply with SGX 2.17. -- File Changes -- M docs/azure-confidential-computing.md (3) -- Patch Links -- https://github.com/apache/incubator-teaclave/pull/687.patch https://github.com/apache/incubator-teaclave/pull/687.diff -- Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-teaclave/pull/687 You are receiving this because you are subscribed to this thread. Message ID: apache/incubator-teaclave/pull/6...@github.com
[apache/incubator-teaclave] Fix build docker files. (PR #686)
## Description `docker build` for `build.ubuntu-1804-sgx-2.17.1.Dockerfile` and `build.ubuntu-1804-sgx-dcap-1.14.Dockerfile` fails due to a lack of Python library dependency. ``` $ docker build -t ubuntu-1804-sgx-2.17.1:latest . -f docker/build.ubuntu-1804-sgx-dcap-1.14.Dockerfile Could not find a version that satisfies the requirement tomli=2.0.1 (from yapf) (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.10, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3) No matching distribution found for tomli=2.0.1 (from yapf) The command /bin/sh -c pip3 install pyopenssl==21.0.0 toml cryptography yapf requests Pillow returned a non-zero code: 1 ``` I confirmed this issue doesnt happen with `build.ubuntu-2004-sgx-2.17.1.Dockerfile` or `build.ubuntu-2004-sgx-dcap-1.14.Dockerfile`. ## Type of change (select or add applied and delete the others) - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] API change with a documentation update - [ ] Additional test coverage - [ ] Code cleanup or just sync with upstream third-party crates ## How has this been tested? ## Checklist - [x] Fork the repo and create your branch from `master`. - [x] If youve added code that should be tested, add tests. - [x] If youve changed APIs, update the documentation. - [x] Ensure the tests pass (see CI results). - [x] Make sure your code lints/format. You can view, comment on, or merge this pull request online at: https://github.com/apache/incubator-teaclave/pull/686 -- Commit Summary -- * Fix docker files. -- File Changes -- M docker/build.ubuntu-1804-sgx-2.17.1.Dockerfile (2) M docker/build.ubuntu-1804-sgx-dcap-1.14.Dockerfile (2) -- Patch Links -- https://github.com/apache/incubator-teaclave/pull/686.patch https://github.com/apache/incubator-teaclave/pull/686.diff -- Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-teaclave/pull/686 You are receiving this because you are subscribed to this thread. Message ID: apache/incubator-teaclave/pull/6...@github.com
[jira] [Created] (AVRO-3741) Note about the version requirement of Rust in BUILD.md
Kousuke Saruta created AVRO-3741: Summary: Note about the version requirement of Rust in BUILD.md Key: AVRO-3741 URL: https://issues.apache.org/jira/browse/AVRO-3741 Project: Apache Avro Issue Type: Improvement Components: doc Affects Versions: 1.12.0 Reporter: Kousuke Saruta The requirement version of Rust is not noted in BUILD.md. It's better to note like other languages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (SPARK-41634) Upgrade minimatch to 3.1.2
[ https://issues.apache.org/jira/browse/SPARK-41634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-41634. Fix Version/s: 3.4.0 Assignee: Bjørn Jørgensen Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/39143 > Upgrade minimatch to 3.1.2 > --- > > Key: SPARK-41634 > URL: https://issues.apache.org/jira/browse/SPARK-41634 > Project: Spark > Issue Type: Dependency upgrade > Components: Build >Affects Versions: 3.4.0 >Reporter: Bjørn Jørgensen >Assignee: Bjørn Jørgensen >Priority: Minor > Fix For: 3.4.0 > > > [CVE-2022-3517|https://nvd.nist.gov/vuln/detail/CVE-2022-3517] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-41587) Upgrade org.scalatestplus:selenium-4-4 to org.scalatestplus:selenium-4-7
[ https://issues.apache.org/jira/browse/SPARK-41587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-41587. Fix Version/s: 3.4.0 Assignee: Yang Jie Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/39129 > Upgrade org.scalatestplus:selenium-4-4 to org.scalatestplus:selenium-4-7 > > > Key: SPARK-41587 > URL: https://issues.apache.org/jira/browse/SPARK-41587 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.4.0 > > > https://github.com/scalatest/scalatestplus-selenium/releases/tag/release-3.2.14.0-for-selenium-4.7 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-40397) Migrate selenium-java from 3.1 to 4.2 and upgrade org.scalatestplus:selenium to 3.2.13.0
[ https://issues.apache.org/jira/browse/SPARK-40397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-40397. Fix Version/s: 3.4.0 Assignee: Yang Jie Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/37868 > Migrate selenium-java from 3.1 to 4.2 and upgrade org.scalatestplus:selenium > to 3.2.13.0 > > > Key: SPARK-40397 > URL: https://issues.apache.org/jira/browse/SPARK-40397 > Project: Spark > Issue Type: Improvement > Components: Build, Tests >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-38303) Upgrade ansi-regex from 5.0.0 to 5.0.1 in /dev
[ https://issues.apache.org/jira/browse/SPARK-38303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-38303. Fix Version/s: 3.3.0 3.2.2 Assignee: Bjørn Jørgensen Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/35628 > Upgrade ansi-regex from 5.0.0 to 5.0.1 in /dev > -- > > Key: SPARK-38303 > URL: https://issues.apache.org/jira/browse/SPARK-38303 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.1, 3.3.0 >Reporter: Bjørn Jørgensen >Assignee: Bjørn Jørgensen >Priority: Major > Fix For: 3.3.0, 3.2.2 > > > [CVE-2021-3807|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3807] > > [releases notes at github|https://github.com/chalk/ansi-regex/releases] > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38303) Upgrade ansi-regex from 5.0.0 to 5.0.1 in /dev
[ https://issues.apache.org/jira/browse/SPARK-38303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-38303: --- Affects Version/s: 3.2.1 > Upgrade ansi-regex from 5.0.0 to 5.0.1 in /dev > -- > > Key: SPARK-38303 > URL: https://issues.apache.org/jira/browse/SPARK-38303 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.1, 3.3.0 >Reporter: Bjørn Jørgensen >Priority: Major > > [CVE-2021-3807|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3807] > > [releases notes at github|https://github.com/chalk/ansi-regex/releases] > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-38278) Add SparkContext.addArchive in PySpark
[
https://issues.apache.org/jira/browse/SPARK-38278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-38278.
Assignee: Hyukjin Kwon
Resolution: Fixed
Issue resolved in https://github.com/apache/spark/pull/35603
> Add SparkContext.addArchive in PySpark
> --
>
> Key: SPARK-38278
> URL: https://issues.apache.org/jira/browse/SPARK-38278
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Assignee: Hyukjin Kwon
>Priority: Major
>
> SPARK-33530 added {{SparkContext.addArchive}} API. We should have one in
> PySpark too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38278) Add SparkContext.addArchive in PySpark
[
https://issues.apache.org/jira/browse/SPARK-38278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-38278:
---
Fix Version/s: 3.3.0
> Add SparkContext.addArchive in PySpark
> --
>
> Key: SPARK-38278
> URL: https://issues.apache.org/jira/browse/SPARK-38278
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Assignee: Hyukjin Kwon
>Priority: Major
> Fix For: 3.3.0
>
>
> SPARK-33530 added {{SparkContext.addArchive}} API. We should have one in
> PySpark too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36808) Upgrade Kafka to 2.8.1
[ https://issues.apache.org/jira/browse/SPARK-36808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36808: --- Fix Version/s: 3.2.2 > Upgrade Kafka to 2.8.1 > -- > > Key: SPARK-36808 > URL: https://issues.apache.org/jira/browse/SPARK-36808 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.2.1, 3.3.0 > Reporter: Kousuke Saruta > Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0, 3.2.2 > > > A few hours ago, Kafka 2.8.1 was released, which includes a bunch of bug fix. > https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36808) Upgrade Kafka to 2.8.1
[ https://issues.apache.org/jira/browse/SPARK-36808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36808: --- Affects Version/s: 3.2.1 > Upgrade Kafka to 2.8.1 > -- > > Key: SPARK-36808 > URL: https://issues.apache.org/jira/browse/SPARK-36808 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.2.1, 3.3.0 > Reporter: Kousuke Saruta > Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0 > > > A few hours ago, Kafka 2.8.1 was released, which includes a bunch of bug fix. > https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36808) Upgrade Kafka to 2.8.1
[ https://issues.apache.org/jira/browse/SPARK-36808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492452#comment-17492452 ] Kousuke Saruta commented on SPARK-36808: Ah, O.K. I misunderstood. I'll withdraw the PRs. > Upgrade Kafka to 2.8.1 > -- > > Key: SPARK-36808 > URL: https://issues.apache.org/jira/browse/SPARK-36808 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta > Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0 > > > A few hours ago, Kafka 2.8.1 was released, which includes a bunch of bug fix. > https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36808) Upgrade Kafka to 2.8.1
[ https://issues.apache.org/jira/browse/SPARK-36808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492408#comment-17492408 ] Kousuke Saruta commented on SPARK-36808: [~dongjoon] Sure, I'll do it. > Upgrade Kafka to 2.8.1 > -- > > Key: SPARK-36808 > URL: https://issues.apache.org/jira/browse/SPARK-36808 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta > Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0 > > > A few hours ago, Kafka 2.8.1 was released, which includes a bunch of bug fix. > https://downloads.apache.org/kafka/2.8.1/RELEASE_NOTES.html -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38149) Upgrade joda-time to 2.10.13
Kousuke Saruta created SPARK-38149: -- Summary: Upgrade joda-time to 2.10.13 Key: SPARK-38149 URL: https://issues.apache.org/jira/browse/SPARK-38149 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.3.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta joda-time 2.10.13 was released, which supports the latest TZ database of 2021e. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37934) Upgrade Jetty version to 9.4.44
[ https://issues.apache.org/jira/browse/SPARK-37934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489017#comment-17489017 ] Kousuke Saruta commented on SPARK-37934: Issue resolved in https://github.com/apache/spark/pull/35442 for branch-3.2. > Upgrade Jetty version to 9.4.44 > --- > > Key: SPARK-37934 > URL: https://issues.apache.org/jira/browse/SPARK-37934 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.2.0, 3.3.0 >Reporter: Sajith A >Assignee: Sajith A >Priority: Minor > Fix For: 3.3.0, 3.2.2 > > > Upgrade Jetty version to 9.4.44.v20210927 in current Spark master to bring-in > the fixes for the > [jetty#6973|https://github.com/eclipse/jetty.project/issues/6973] issue. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37934) Upgrade Jetty version to 9.4.44
[ https://issues.apache.org/jira/browse/SPARK-37934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-37934: --- Fix Version/s: 3.2.2 > Upgrade Jetty version to 9.4.44 > --- > > Key: SPARK-37934 > URL: https://issues.apache.org/jira/browse/SPARK-37934 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.2.0, 3.3.0 >Reporter: Sajith A >Assignee: Sajith A >Priority: Minor > Fix For: 3.3.0, 3.2.2 > > > Upgrade Jetty version to 9.4.44.v20210927 in current Spark master to bring-in > the fixes for the > [jetty#6973|https://github.com/eclipse/jetty.project/issues/6973] issue. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38087) select doesnt validate if the column already exists
[
https://issues.apache.org/jira/browse/SPARK-38087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-38087:
---
Component/s: SQL
(was: Spark Core)
> select doesnt validate if the column already exists
> ---
>
> Key: SPARK-38087
> URL: https://issues.apache.org/jira/browse/SPARK-38087
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.1
> Environment: Version{{{}v3.2.1{}}}
> {{}}
> {{{}{}}}Master{{{}local[*]{}}}
> {{(Reproducible in any environment)}}
>Reporter: Deepa Vasanthkumar
>Priority: Minor
> Attachments: select vs drop.png
>
>
>
> Select doesnt validate whether the alias column is already present in the
> dataframe.
> After which, we cannot do anything in that dataframe on that column.
> df4 = df2.select(df2.firstname, df2.lastname) --> throws analysis exception
> df4.show()
>
> However drop will not let you drop the said column.
>
> Scenario to reproduce :
> df2 = df1.select("*", (df1.firstname).alias("firstname")) ---> this will
> add same column
> df2.show()
> df2.drop(df2.firstname) --> this will give AnalysisException: Reference
> 'firstname' is ambiguous, could be: firstname, firstname.
>
>
> Is this expected behavior .
> !select vs drop.png!
> !image-2022-02-02-06-28-23-543.png!
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-38021) Upgrade dropwizard metrics from 4.2.2 to 4.2.7
[ https://issues.apache.org/jira/browse/SPARK-38021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-38021. Fix Version/s: 3.3.0 Assignee: Yang Jie Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/35317. > Upgrade dropwizard metrics from 4.2.2 to 4.2.7 > -- > > Key: SPARK-38021 > URL: https://issues.apache.org/jira/browse/SPARK-38021 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Minor > Fix For: 3.3.0 > > > dropwizard metrics has released 5 versions after 4.2.2： > * [https://github.com/dropwizard/metrics/releases/tag/v4.2.3] > * [https://github.com/dropwizard/metrics/releases/tag/v4.2.4] > * [https://github.com/dropwizard/metrics/releases/tag/v4.2.5] > * [https://github.com/dropwizard/metrics/releases/tag/v4.2.6] > * [https://github.com/dropwizard/metrics/releases/tag/v4.2.7] > > And after 4.2.5 version, codahale metrics supports build with JDK 17 > (https://github.com/dropwizard/metrics/pull/2180) > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-38017) Fix the API doc for window to say it supports TimestampNTZType too as timeColumn
[ https://issues.apache.org/jira/browse/SPARK-38017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-38017. Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/35313. > Fix the API doc for window to say it supports TimestampNTZType too as > timeColumn > > > Key: SPARK-38017 > URL: https://issues.apache.org/jira/browse/SPARK-38017 > Project: Spark > Issue Type: Bug > Components: Documentation, SQL >Affects Versions: 3.2.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Fix For: 3.3.0, 3.2.2 > > > window function supports not only TimestampType but also TimestampNTZType but > the API docs doesn't mention TimestampNTZType. > This issue is similar to SPARK-38016, but this issue affects 3.2.0 too, so I > separate the tickets. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-38016) Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn
[ https://issues.apache.org/jira/browse/SPARK-38016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-38016. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/35312. > Fix the API doc for session_window to say it supports TimestampNTZType too as > timeColumn > > > Key: SPARK-38016 > URL: https://issues.apache.org/jira/browse/SPARK-38016 > Project: Spark > Issue Type: Bug > Components: Documentation, SQL >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > Fix For: 3.3.0 > > > As of Spark 3.3.0, session_window supports not only TimestampType but also > TimestampNTZType but the API docs doesn't mention TimestampNTZType. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38016) Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn
[ https://issues.apache.org/jira/browse/SPARK-38016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-38016: --- Description: As of Spark 3.3.0, session_window supports not only TimestampType but also TimestampNTZType but the API docs doesn't mention TimestampNTZType. (was: As of Spark 3.3.0, session_window supports not only TimestampType but also TimestampNTZType but the API docs mention TimestampNTZType.) > Fix the API doc for session_window to say it supports TimestampNTZType too as > timeColumn > > > Key: SPARK-38016 > URL: https://issues.apache.org/jira/browse/SPARK-38016 > Project: Spark > Issue Type: Bug > Components: Documentation, SQL >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > > As of Spark 3.3.0, session_window supports not only TimestampType but also > TimestampNTZType but the API docs doesn't mention TimestampNTZType. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38017) Fix the API doc for window to say it supports TimestampNTZType too as timeColumn
[ https://issues.apache.org/jira/browse/SPARK-38017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-38017: --- Description: window function supports not only TimestampType but also TimestampNTZType but the API docs doesn't mention TimestampNTZType. This issue is similar to SPARK-38016, but this issue affects 3.2.0 too, so I separate the tickets. was: window function supports not only TimestampType but also TimestampNTZType but the API docs mention TimestampNTZType. This issue is similar to SPARK-38016, but this issue affects 3.2.0 too, so I separate the tickets. > Fix the API doc for window to say it supports TimestampNTZType too as > timeColumn > > > Key: SPARK-38017 > URL: https://issues.apache.org/jira/browse/SPARK-38017 > Project: Spark > Issue Type: Bug > Components: Documentation, SQL >Affects Versions: 3.2.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > > window function supports not only TimestampType but also TimestampNTZType but > the API docs doesn't mention TimestampNTZType. > This issue is similar to SPARK-38016, but this issue affects 3.2.0 too, so I > separate the tickets. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38017) Fix the API doc for window to say it supports TimestampNTZType too as timeColumn
Kousuke Saruta created SPARK-38017: -- Summary: Fix the API doc for window to say it supports TimestampNTZType too as timeColumn Key: SPARK-38017 URL: https://issues.apache.org/jira/browse/SPARK-38017 Project: Spark Issue Type: Bug Components: Documentation, SQL Affects Versions: 3.2.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta window function supports not only TimestampType but also TimestampNTZType but the API docs mention TimestampNTZType. This issue is similar to SPARK-38016, but this issue affects 3.2.0 too, so I separate the tickets. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38016) Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn
[ https://issues.apache.org/jira/browse/SPARK-38016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-38016: --- Summary: Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn (was: Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn.) > Fix the API doc for session_window to say it supports TimestampNTZType too as > timeColumn > > > Key: SPARK-38016 > URL: https://issues.apache.org/jira/browse/SPARK-38016 > Project: Spark > Issue Type: Bug > Components: Documentation, SQL >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > > As of Spark 3.3.0, session_window supports not only TimestampType but also > TimestampNTZType but the API docs mention TimestampNTZType. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38016) Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn.
[ https://issues.apache.org/jira/browse/SPARK-38016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-38016: --- Summary: Fix the API doc for session_window to say it supports TimestampNTZType too as timeColumn. (was: Fix the API doc for window and session_window to say it supports TimestampNTZType too as timeColumn.) > Fix the API doc for session_window to say it supports TimestampNTZType too as > timeColumn. > - > > Key: SPARK-38016 > URL: https://issues.apache.org/jira/browse/SPARK-38016 > Project: Spark > Issue Type: Bug > Components: Documentation, SQL >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > > As of Spark 3.3.0, session_window supports not only TimestampType but also > TimestampNTZType but the API docs mention TimestampNTZType. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38016) Fix the API doc for window and session_window to say it supports TimestampNTZType too as timeColumn.
Kousuke Saruta created SPARK-38016: -- Summary: Fix the API doc for window and session_window to say it supports TimestampNTZType too as timeColumn. Key: SPARK-38016 URL: https://issues.apache.org/jira/browse/SPARK-38016 Project: Spark Issue Type: Bug Components: Documentation, SQL Affects Versions: 3.3.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta As of Spark 3.3.0, session_window supports not only TimestampType but also TimestampNTZType but the API docs mention TimestampNTZType. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37860) [BUG] Revert: Fix taskid in the stage page task event timeline
[
https://issues.apache.org/jira/browse/SPARK-37860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472487#comment-17472487
]
Kousuke Saruta commented on SPARK-37860:
Note: If the vote of Spark 3.2.1 RC1 passes, replace the fix version of 3.2.1
with 3.2.2.
> [BUG] Revert: Fix taskid in the stage page task event timeline
> --
>
> Key: SPARK-37860
> URL: https://issues.apache.org/jira/browse/SPARK-37860
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.2.1
>Reporter: Jackey Lee
>Assignee: Jackey Lee
>Priority: Major
> Fix For: 3.1.3, 3.0.4, 3.2.1, 3.3.0
>
>
> In [#32888|https://github.com/apache/spark/pull/32888],
> [@shahidki31|https://github.com/shahidki31] change taskInfo.index to
> taskInfo.taskId. However, we generally use {{index.attempt}} or {{taskId}} to
> distinguish tasks within a stage, not {{{}taskId.attempt{}}}.
> Thus [#32888|https://github.com/apache/spark/pull/32888] was a wrong fix
> issue, we should revert it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37860) [BUG] Revert: Fix taskid in the stage page task event timeline
[
https://issues.apache.org/jira/browse/SPARK-37860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-37860.
Fix Version/s: 3.1.3
3.0.4
3.2.1
3.3.0
Assignee: Jackey Lee
Resolution: Fixed
Issue resolved in https://github.com/apache/spark/pull/35160
> [BUG] Revert: Fix taskid in the stage page task event timeline
> --
>
> Key: SPARK-37860
> URL: https://issues.apache.org/jira/browse/SPARK-37860
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 3.2.1
>Reporter: Jackey Lee
>Assignee: Jackey Lee
>Priority: Major
> Fix For: 3.1.3, 3.0.4, 3.2.1, 3.3.0
>
>
> In [#32888|https://github.com/apache/spark/pull/32888],
> [@shahidki31|https://github.com/shahidki31] change taskInfo.index to
> taskInfo.taskId. However, we generally use {{index.attempt}} or {{taskId}} to
> distinguish tasks within a stage, not {{{}taskId.attempt{}}}.
> Thus [#32888|https://github.com/apache/spark/pull/32888] was a wrong fix
> issue, we should revert it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37159) Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
[
https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472434#comment-17472434
]
Kousuke Saruta commented on SPARK-37159:
All right. Thank you [~dongjoon]!
> Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
> ---
>
> Key: SPARK-37159
> URL: https://issues.apache.org/jira/browse/SPARK-37159
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Tests
>Affects Versions: 3.3.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
>Priority: Minor
> Fix For: 3.3.0
>
>
> SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but
> `HiveExternalCatalogVersionsSuite`.
> {code}
> [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED
> *** (42 seconds, 526 milliseconds)
> [info] spark-submit returned with exit code 1.
> [info] Command line:
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit'
> '--name' 'prepare testing tables' '--master' 'local[2]' '--conf'
> 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf'
> 'spark.sql.hive.metastore.version=2.3' '--conf'
> 'spark.sql.hive.metastore.jars=maven' '--conf'
> 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
> '--conf' 'spark.sql.test.version.index=0' '--driver-java-options'
> '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62'
>
> '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py'
> [info]
> [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Running Spark version 3.2.0
> [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicable
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: No custom resources configured for spark.driver.
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO
> ResourceUtils: ==
> [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO
> SparkContext: Submitted application: prepare testing tables
> [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Default ResourceProfile created, executor resources:
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name:
> memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount:
> 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfile: Limiting resource is cpu
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO
> ResourceProfileManager: Added ResourceProfile id: 0
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls to: kou
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls to: kou
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing view acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: Changing modify acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO
> SecurityManager: SecurityManager: authentication disabled; ui acls disabled;
> users with view permissions: Set(kou); groups with view permissions: Set();
> users with modify permissions: Set(kou); groups with modify permissions:
> Set()
> [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils:
> Successfully started service 'sparkDriver' on port 35867.
> [info]
[jira] [Resolved] (SPARK-37792) Spark shell sets log level to INFO by default
[
https://issues.apache.org/jira/browse/SPARK-37792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta resolved SPARK-37792.
Fix Version/s: 3.3.0
Assignee: L. C. Hsieh (was: Apache Spark)
Resolution: Fixed
Issue resolved in https://github.com/apache/spark/pull/35080
> Spark shell sets log level to INFO by default
> -
>
> Key: SPARK-37792
> URL: https://issues.apache.org/jira/browse/SPARK-37792
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
>Affects Versions: 3.3.0
>Reporter: Hyukjin Kwon
>Assignee: L. C. Hsieh
>Priority: Major
> Fix For: 3.3.0
>
>
> {code}
> ./bin/spark-shell
> {code}
> {code}
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 21/12/31 10:55:04 INFO SignalUtils: Registering signal handler for INT
> 21/12/31 10:55:08 INFO HiveConf: Found configuration file null
> 21/12/31 10:55:08 INFO SparkContext: Running Spark version 3.3.0-SNAPSHOT
> ...
> 21/12/31 10:55:09 INFO BlockManager: Initialized BlockManager:
> BlockManagerId(driver, ..., None)
> ...
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
>/___/ .__/\_,_/_/ /_/\_\ version 3.3.0-SNAPSHOT
> /_/
> Using Scala version 2.12.15 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_291)
> Type in expressions to have them evaluated.
> Type :help for more information.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37778) Upgrade SBT to 1.6.1
Kousuke Saruta created SPARK-37778: -- Summary: Upgrade SBT to 1.6.1 Key: SPARK-37778 URL: https://issues.apache.org/jira/browse/SPARK-37778 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.3.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta SBT 1.6.1 was released, which log4j 2 to 2.17.1 for CVE-2021-44832. https://github.com/sbt/sbt/releases/tag/v1.6.1 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37391) SIGNIFICANT bottleneck introduced by fix for SPARK-32001
[ https://issues.apache.org/jira/browse/SPARK-37391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-37391. Fix Version/s: 3.3.0 Assignee: Danny Guinther Resolution: Fixed Issue resolved in https://github.com/apache/spark/pull/34745 for Spark 3.3.0. > SIGNIFICANT bottleneck introduced by fix for SPARK-32001 > > > Key: SPARK-37391 > URL: https://issues.apache.org/jira/browse/SPARK-37391 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0, 3.1.1, 3.1.2, 3.2.0 > Environment: N/A >Reporter: Danny Guinther >Assignee: Danny Guinther >Priority: Major > Fix For: 3.3.0 > > Attachments: so-much-blocking.jpg, spark-regression-dashes.jpg > > > The fix for https://issues.apache.org/jira/browse/SPARK-32001 ( > [https://github.com/apache/spark/pull/29024/files#diff-345beef18081272d77d91eeca2d9b5534ff6e642245352f40f4e9c9b8922b085R58] > ) does not seem to have consider the reality that some apps may rely on > being able to establish many JDBC connections simultaneously for performance > reasons. > The fix forces concurrency to 1 when establishing database connections and > that strikes me as a *significant* user impacting change and a *significant* > bottleneck. > Can anyone propose a workaround for this? I have an app that makes > connections to thousands of databases and I can't upgrade to any version > >3.1.x because of this significant bottleneck. > > Thanks in advance for your help! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37663) Mitigate ConcurrentModificationException thrown from tests in SparkContextSuite
[ https://issues.apache.org/jira/browse/SPARK-37663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-37663: --- Summary: Mitigate ConcurrentModificationException thrown from tests in SparkContextSuite (was: Mitigate ConcurrentModificationException thrown from a test in SparkContextSuite) > Mitigate ConcurrentModificationException thrown from tests in > SparkContextSuite > --- > > Key: SPARK-37663 > URL: https://issues.apache.org/jira/browse/SPARK-37663 > Project: Spark > Issue Type: Bug > Components: Spark Core, Tests >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > > ConcurrentModificationException can be thrown from tests in SparkContextSuite > with Scala 2.13. > The cause seems to be same as SPARK-37315. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37663) SPARK-37315][ML][TEST] Mitigate ConcurrentModificationException thrown from a test in SparkContextSuite
Kousuke Saruta created SPARK-37663: -- Summary: SPARK-37315][ML][TEST] Mitigate ConcurrentModificationException thrown from a test in SparkContextSuite Key: SPARK-37663 URL: https://issues.apache.org/jira/browse/SPARK-37663 Project: Spark Issue Type: Bug Components: Spark Core, Tests Affects Versions: 3.3.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta ConcurrentModificationException can be thrown from tests in SparkContextSuite with Scala 2.13. The cause seems to be same as SPARK-37315. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37663) Mitigate ConcurrentModificationException thrown from a test in SparkContextSuite
[ https://issues.apache.org/jira/browse/SPARK-37663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-37663: --- Summary: Mitigate ConcurrentModificationException thrown from a test in SparkContextSuite (was: SPARK-37315][ML][TEST] Mitigate ConcurrentModificationException thrown from a test in SparkContextSuite) > Mitigate ConcurrentModificationException thrown from a test in > SparkContextSuite > > > Key: SPARK-37663 > URL: https://issues.apache.org/jira/browse/SPARK-37663 > Project: Spark > Issue Type: Bug > Components: Spark Core, Tests >Affects Versions: 3.3.0 > Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Minor > > ConcurrentModificationException can be thrown from tests in SparkContextSuite > with Scala 2.13. > The cause seems to be same as SPARK-37315. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37656) Upgrade SBT to 1.5.7
Kousuke Saruta created SPARK-37656: -- Summary: Upgrade SBT to 1.5.7 Key: SPARK-37656 URL: https://issues.apache.org/jira/browse/SPARK-37656 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.2.1, 3.3.0 Reporter: Kousuke Saruta Assignee: Kousuke Saruta SBT 1.5.7 was released a few hours ago, which includes a fix for CVE-2021-45046. https://github.com/sbt/sbt/releases/tag/v1.5.7 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
