Change in asterixdb[master]: [ASTERIXDB-3343][API] Include job details in active/completed requests
>From Jenkins : Attention is currently required from: Wail Alkowaileet, Ali Alsuliman, Michael Blow. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18148 ) Change subject: [ASTERIXDB-3343][API] Include job details in active/completed requests .. Patch Set 6: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/330/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18148 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id1bdc935be8bf84674aa6e35ee6c19f2ee1f7971 Gerrit-Change-Number: 18148 Gerrit-PatchSet: 6 Gerrit-Owner: Ali Alsuliman Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Ali Alsuliman Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Fri, 19 Apr 2024 23:23:11 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3343][API] Include job details in active/completed requests
>From Murtadha Hubail : Attention is currently required from: Wail Alkowaileet, Ali Alsuliman, Michael Blow. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18148 ) Change subject: [ASTERIXDB-3343][API] Include job details in active/completed requests .. Patch Set 6: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18148 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id1bdc935be8bf84674aa6e35ee6c19f2ee1f7971 Gerrit-Change-Number: 18148 Gerrit-PatchSet: 6 Gerrit-Owner: Ali Alsuliman Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Ali Alsuliman Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Fri, 19 Apr 2024 22:46:17 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3343][API] Include job details in active/completed requests
>From Ali Alsuliman : Attention is currently required from: Wail Alkowaileet, Ali Alsuliman, Michael Blow. Hello Murtadha Hubail, Wail Alkowaileet, Jenkins, Michael Blow, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18148 to look at the new patch set (#6). Change subject: [ASTERIXDB-3343][API] Include job details in active/completed requests .. [ASTERIXDB-3343][API] Include job details in active/completed requests - user model changes: no - storage format changes: no - interface changes: yes Details: Include job status, create time, start time, queue time, and end time. Change-Id: Id1bdc935be8bf84674aa6e35ee6c19f2ee1f7971 --- M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IReceptionist.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/job/JobRun.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/job/profiling/om/JobProfile.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/Receptionist.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/BaseClientRequest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/QueryCancellationServletTest.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/JobResultCallback.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/ClientRequest.java 9 files changed, 123 insertions(+), 37 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/48/18148/6 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18148 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id1bdc935be8bf84674aa6e35ee6c19f2ee1f7971 Gerrit-Change-Number: 18148 Gerrit-PatchSet: 6 Gerrit-Owner: Ali Alsuliman Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Ali Alsuliman Gerrit-Attention: Michael Blow Gerrit-MessageType: newpatchset
[jira] [Commented] (ASTERIXDB-3375) Introduce local disk caching APIs for cloud deployment
[ https://issues.apache.org/jira/browse/ASTERIXDB-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839139#comment-17839139 ] ASF subversion and git services commented on ASTERIXDB-3375: Commit 436b78e7d01c24a36d02e51502ce23a8cbddcfad in asterixdb's branch refs/heads/master from Wail Alkowaileet [ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=436b78e7d0 ] [ASTERIXDB-3375][STO][HYR] Introduce local disk caching APIs - user model changes: no - storage format changes: no - interface changes: no Details: Introduce hyracks-cloud and local disk caching APIs Change-Id: I8a1ce6616d1190f8b047570fac1d774297a3d7de Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227 Integration-Tests: Jenkins Tested-by: Jenkins Reviewed-by: Wail Alkowaileet Reviewed-by: Murtadha Hubail > Introduce local disk caching APIs for cloud deployment > -- > > Key: ASTERIXDB-3375 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3375 > Project: Apache AsterixDB > Issue Type: Improvement > Components: HYR - Hyracks, STO - Storage >Affects Versions: 0.9.10 >Reporter: Wail Y. Alkowaileet >Assignee: Wail Y. Alkowaileet >Priority: Major > Labels: triaged > Fix For: 0.9.10 > > > Introduce hyracks-cloud module along with local disk caching APIs -- This message was sent by Atlassian Jira (v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3375][STO][HYR] Introduce local disk caching APIs
>From Murtadha Hubail : Attention is currently required from: Wail Alkowaileet. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227 ) Change subject: [ASTERIXDB-3375][STO][HYR] Introduce local disk caching APIs .. Patch Set 4: Code-Review+2 (1 comment) File hyracks-fullstack/hyracks/hyracks-cloud/src/main/java/org/apache/hyracks/cloud/buffercache/context/ContextUtil.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227/comment/3239b366_bdc6a625 PS4, Line 30: ContextUtil Add identifier before the Context -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I8a1ce6616d1190f8b047570fac1d774297a3d7de Gerrit-Change-Number: 18227 Gerrit-PatchSet: 4 Gerrit-Owner: Wail Alkowaileet Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Wail Alkowaileet Gerrit-Comment-Date: Fri, 19 Apr 2024 22:38:11 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
[jira] [Resolved] (ASTERIXDB-3385) Duplicate fields should not accepted in primary key
[
https://issues.apache.org/jira/browse/ASTERIXDB-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peeyush Gupta resolved ASTERIXDB-3385.
--
Resolution: Fixed
> Duplicate fields should not accepted in primary key
> ---
>
> Key: ASTERIXDB-3385
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3385
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: COMP - Compiler
>Reporter: Peeyush Gupta
>Priority: Major
>
> The following create dataset statement should not be allowed.
> {noformat}
> CREATE DATASET test PRIMARY KEY(name: string, name:string);{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (ASTERIXDB-3385) Duplicate fields should not accepted in primary key
[
https://issues.apache.org/jira/browse/ASTERIXDB-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839134#comment-17839134
]
ASF subversion and git services commented on ASTERIXDB-3385:
Commit a1ba5ae8467bf2dfea6255bb759e6dece53c487a in asterixdb's branch
refs/heads/master from Peeyush Gupta
[ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=a1ba5ae846 ]
[ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key
- user model changes: no
- storage format changes: no
- interface changes: no
Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244
Integration-Tests: Jenkins
Reviewed-by: Murtadha Hubail
Tested-by: Peeyush Gupta
> Duplicate fields should not accepted in primary key
> ---
>
> Key: ASTERIXDB-3385
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3385
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: COMP - Compiler
>Reporter: Peeyush Gupta
>Priority: Major
>
> The following create dataset statement should not be allowed.
> {noformat}
> CREATE DATASET test PRIMARY KEY(name: string, name:string);{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primar...
>From Peeyush Gupta :
Peeyush Gupta has submitted this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 )
Change subject: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in
primary key
..
[ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key
- user model changes: no
- storage format changes: no
- interface changes: no
Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244
Integration-Tests: Jenkins
Reviewed-by: Murtadha Hubail
Tested-by: Peeyush Gupta
---
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.04.ddl.sqlpp
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
6 files changed, 97 insertions(+), 0 deletions(-)
Approvals:
Murtadha Hubail: Looks good to me, approved
Peeyush Gupta: Verified
Jenkins: Verified
diff --git
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
index e253e07..34976b6 100644
---
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
+++
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
@@ -122,10 +122,36 @@
autogenerated, true, sourceLoc, partitioningExprTypes);
}
+private static void checkDuplicateFields(List>
partitioningExprs, SourceLocation sourceLoc)
+throws CompilationException {
+for (int i = 0; i < partitioningExprs.size(); i++) {
+List expr1 = partitioningExprs.get(i);
+for (int j = i + 1; j < partitioningExprs.size(); j++) {
+List expr2 = partitioningExprs.get(j);
+int k = 0;
+while (k < expr1.size() && k < expr2.size()) {
+if (!expr1.get(k).equals(expr2.get(k))) {
+break;
+} else {
+k += 1;
+}
+}
+if (k == expr1.size() && k == expr2.size()) {
+throw new
CompilationException(ErrorCode.DUPLICATE_FIELD_IN_PRIMARY_KEY, sourceLoc,
+String.join(".", expr1));
+} else if (k == expr1.size() || k == expr2.size()) {
+throw new
CompilationException(ErrorCode.INCOMPATIBLE_FIELDS_IN_PRIMARY_KEY, sourceLoc,
+String.join(".", expr1), String.join(".", expr2));
+}
+}
+}
+}
+
private static List
validatePartitioningExpressionsImpl(ARecordType recType, ARecordType
metaRecType,
List> partitioningExprs, List
keySourceIndicators, boolean autogenerated,
boolean forPrimaryKey, SourceLocation sourceLoc,
List partitioningExprTypes)
throws AlgebricksException {
+checkDuplicateFields(partitioningExprs, sourceLoc);
String keyKindDisplayName = forPrimaryKey ? PRIMARY : "";
List computedPartitioningExprTypes = new
ArrayList<>(partitioningExprs.size());
if (autogenerated) {
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
new file mode 100644
index 000..52c5119
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing
Change in asterixdb[master]: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primar...
>From Peeyush Gupta : Attention is currently required from: Ali Alsuliman. Peeyush Gupta has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 ) Change subject: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key .. Patch Set 2: Verified+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 Gerrit-Change-Number: 18244 Gerrit-PatchSet: 2 Gerrit-Owner: Peeyush Gupta Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Ali Alsuliman Gerrit-Comment-Date: Fri, 19 Apr 2024 22:21:43 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primar...
>From Peeyush Gupta : Attention is currently required from: Peeyush Gupta, Ali Alsuliman. Peeyush Gupta has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 ) Change subject: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key .. Removed Verified-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 Gerrit-Change-Number: 18244 Gerrit-PatchSet: 2 Gerrit-Owner: Peeyush Gupta Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Ali Alsuliman Gerrit-MessageType: deleteVote
Change in asterixdb[master]: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primar...
>From Murtadha Hubail : Attention is currently required from: Peeyush Gupta, Ali Alsuliman. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 ) Change subject: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key .. Patch Set 2: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 Gerrit-Change-Number: 18244 Gerrit-PatchSet: 2 Gerrit-Owner: Peeyush Gupta Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Ali Alsuliman Gerrit-Comment-Date: Fri, 19 Apr 2024 21:42:25 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
[jira] [Commented] (ASTERIXDB-3383) Add tuple filter to ingestion pipeline
[ https://issues.apache.org/jira/browse/ASTERIXDB-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839126#comment-17839126 ] ASF subversion and git services commented on ASTERIXDB-3383: Commit 414ea84d87b9fc4952d40fdcdf3f192694cce07e in asterixdb's branch refs/heads/master from Ritik Raj [ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=414ea84d87 ] [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline - user model changes: no - storage format changes: no - interface changes: no Details: - excludes the tuple from being processed based on the provided filter. Change-Id: Id79607bdada1cd42949cccf43390a90dda092602 Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 Integration-Tests: Jenkins Reviewed-by: Murtadha Hubail Tested-by: Murtadha Hubail > Add tuple filter to ingestion pipeline > -- > > Key: ASTERIXDB-3383 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3383 > Project: Apache AsterixDB > Issue Type: Improvement > Components: RT - Runtime >Affects Versions: 0.9.10 >Reporter: Murtadha Makki Al Hubail >Assignee: Ritik Raj >Priority: Major > Labels: triaged > Fix For: 0.9.10 > > > Add a tuple filter that can be used to exclude tuple from being processed by > the ingestion pipeline. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline
>From Murtadha Hubail : Attention is currently required from: Peeyush Gupta, Ritik Raj. Murtadha Hubail has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 ) Change subject: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline .. Removed Verified-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id79607bdada1cd42949cccf43390a90dda092602 Gerrit-Change-Number: 18239 Gerrit-PatchSet: 3 Gerrit-Owner: Ritik Raj Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Ritik Raj Gerrit-MessageType: deleteVote
Change in asterixdb[master]: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline
>From Murtadha Hubail :
Murtadha Hubail has submitted this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 )
Change subject: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion
pipeline
..
[ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- excludes the tuple from being processed
based on the provided filter.
Change-Id: Id79607bdada1cd42949cccf43390a90dda092602
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239
Integration-Tests: Jenkins
Reviewed-by: Murtadha Hubail
Tested-by: Murtadha Hubail
---
A
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/NoOpLSMTupleFilterCallbackFactory.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java
A
hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMTupleFilterCallback.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMPrimaryUpsertOperatorDescriptor.java
A
hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMTupleFilterCallbackFactory.java
A
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/NoOpLSMTupleFilterCallback.java
M
asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMPrimaryUpsertOperatorNodePushable.java
9 files changed, 198 insertions(+), 6 deletions(-)
Approvals:
Murtadha Hubail: Looks good to me, approved; Verified
Jenkins: Verified
diff --git
a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
index fc992e3..668ff13 100644
---
a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
+++
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java
@@ -52,6 +52,7 @@
import org.apache.asterix.metadata.entities.Dataset;
import org.apache.asterix.metadata.entities.Dataverse;
import org.apache.asterix.metadata.entities.Index;
+import org.apache.asterix.metadata.entities.NoOpLSMTupleFilterCallbackFactory;
import org.apache.asterix.metadata.utils.DatasetUtil;
import org.apache.asterix.metadata.utils.SplitsAndConstraintsUtil;
import org.apache.asterix.om.types.ARecordType;
@@ -843,7 +844,7 @@
frameOpCallbackFactory == null ?
dataset.getFrameOpCallbackFactory(mdProvider)
: frameOpCallbackFactory,
MissingWriterFactory.INSTANCE, hasSecondaries,
NoOpTupleProjectorFactory.INSTANCE,
-tuplePartitionerFactory, partitionsMap);
+tuplePartitionerFactory, partitionsMap,
NoOpLSMTupleFilterCallbackFactory.INSTANCE);
RecordDescriptor upsertOutRecDesc =
getUpsertOutRecDesc(primaryIndexInfo.rDesc, dataset,
filterFields == null ? 0 : filterFields.length, recordType,
metaType);
// fix pk fields
diff --git
a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java
b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java
index 85fbad1..9d41097 100644
---
a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java
+++
b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java
@@ -116,6 +116,7 @@
import org.apache.hyracks.storage.am.lsm.common.api.ILSMMergePolicyFactory;
import
org.apache.hyracks.storage.am.lsm.common.api.ILSMOperationTrackerFactory;
import
org.apache.hyracks.storage.am.lsm.common.api.ILSMPageWriteCallbackFactory;
+import
org.apache.hyracks.storage.am.lsm.common.api.ILSMTupleFilterCallbackFactory;
import org.apache.hyracks.storage.common.IResourceFactory;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
@@ -877,4 +878,8 @@
public DatasetFormatInfo getDatasetFormatInfo() {
return datasetFormatInfo;
}
+
+public ILSMTupleFilterCallbackFactory getTupleFilterCallbackFactory() {
+return NoOpLSMTupleFilterCallbackFactory.INSTANCE;
+}
}
diff --git
a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/NoOpLSMTupleFilterCallback.java
b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/NoOpLSMTupleFilterCallback.java
new file mode 100644
index 000..0957a96
--- /dev/null
+++
Change in asterixdb[master]: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline
>From Murtadha Hubail : Attention is currently required from: Peeyush Gupta, Ritik Raj. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 ) Change subject: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline .. Patch Set 3: Verified+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id79607bdada1cd42949cccf43390a90dda092602 Gerrit-Change-Number: 18239 Gerrit-PatchSet: 3 Gerrit-Owner: Ritik Raj Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Ritik Raj Gerrit-Comment-Date: Fri, 19 Apr 2024 21:27:38 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline
>From Murtadha Hubail : Attention is currently required from: Peeyush Gupta, Ritik Raj. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 ) Change subject: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline .. Patch Set 3: (1 comment) Commit Message: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239/comment/53ff55f0_4cdb3a9e PS3, Line 9: The spacing isn't right -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id79607bdada1cd42949cccf43390a90dda092602 Gerrit-Change-Number: 18239 Gerrit-PatchSet: 3 Gerrit-Owner: Ritik Raj Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Ritik Raj Gerrit-Comment-Date: Fri, 19 Apr 2024 21:24:58 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline
>From Murtadha Hubail : Attention is currently required from: Peeyush Gupta, Ritik Raj. Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 ) Change subject: [ASTERIXDB-3383][RT] Introducing tuple filter to ingestion pipeline .. Patch Set 3: Code-Review+2 (5 comments) Patchset: PS3: Feel free to address the comments in another patch. File asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239/comment/0c8e5726_c771251c PS3, Line 536: // get the Tuple filter callback You really don't need this comment. File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMTupleFilterCallback.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239/comment/3764cba8_9d9aec65 PS3, Line 29: Populates the state within the filter callback by fetching the state from the provided "index" : * and mapping it with the storage partition index, which will be utilized for filtering : * the tuple directed to the index "storagePartitionIdx" You should just replace this by "initializes the filter callback" or something generic like that. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239/comment/f4668510_d8e7a4bd PS3, Line 36: Filter the received record based on the initialized ingestion state. This should be more generic than this. It isn't really tired to ingestion. You can see other examples using javadocs in the codebase. File hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMTupleFilterCallbackFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239/comment/3e97005a_a09f1916 PS3, Line 27: * The field tuple contains information about the position of the record, meta, and primary key, : * facilitating the extraction of relevant information for filtering purposes. Remove this. It is too specific to one implementation. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18239 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Id79607bdada1cd42949cccf43390a90dda092602 Gerrit-Change-Number: 18239 Gerrit-PatchSet: 3 Gerrit-Owner: Ritik Raj Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Ritik Raj Gerrit-Comment-Date: Fri, 19 Apr 2024 21:24:06 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primar...
>From Jenkins : Attention is currently required from: Murtadha Hubail, Ali Alsuliman. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 ) Change subject: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key .. Patch Set 2: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/329/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 Gerrit-Change-Number: 18244 Gerrit-PatchSet: 2 Gerrit-Owner: Peeyush Gupta Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Attention: Murtadha Hubail Gerrit-Attention: Ali Alsuliman Gerrit-Comment-Date: Fri, 19 Apr 2024 20:30:52 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primar...
>From Peeyush Gupta : Attention is currently required from: Peeyush Gupta. Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 to look at the new patch set (#2). Change subject: [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key .. [ASTERIXDB-3385][COMP] Duplicate fields should not accepted in primary key - user model changes: no - storage format changes: no - interface changes: no Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 --- M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java M asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.04.ddl.sqlpp M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp 6 files changed, 93 insertions(+), 0 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/44/18244/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 Gerrit-Change-Number: 18244 Gerrit-PatchSet: 2 Gerrit-Owner: Peeyush Gupta Gerrit-Reviewer: Jenkins Gerrit-Attention: Peeyush Gupta Gerrit-MessageType: newpatchset
[jira] [Created] (ASTERIXDB-3385) Duplicate fields should not accepted in primary key
Peeyush Gupta created ASTERIXDB-3385:
Summary: Duplicate fields should not accepted in primary key
Key: ASTERIXDB-3385
URL: https://issues.apache.org/jira/browse/ASTERIXDB-3385
Project: Apache AsterixDB
Issue Type: Bug
Components: COMP - Compiler
Reporter: Peeyush Gupta
The following create dataset statement should not be allowed.
{noformat}
CREATE DATASET test PRIMARY KEY(name: string, name:string);{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3375][STO][HYR] Introduce local disk caching APIs
>From Wail Alkowaileet : Attention is currently required from: Murtadha Hubail. Wail Alkowaileet has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227 ) Change subject: [ASTERIXDB-3375][STO][HYR] Introduce local disk caching APIs .. Patch Set 4: Code-Review+1 (1 comment) File hyracks-fullstack/hyracks/hyracks-cloud/src/main/java/org/apache/hyracks/cloud/cache/service/CloudDiskCacheMonitoringAndPrefetchingService.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227/comment/44a4055e_4be43d30 PS4, Line 63: request We might not need prefetching at all. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18227 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I8a1ce6616d1190f8b047570fac1d774297a3d7de Gerrit-Change-Number: 18227 Gerrit-PatchSet: 4 Gerrit-Owner: Wail Alkowaileet Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Fri, 19 Apr 2024 17:22:07 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
[jira] [Updated] (ASTERIXDB-3384) COPY (IN)TO Missing docs
[ https://issues.apache.org/jira/browse/ASTERIXDB-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Maxon updated ASTERIXDB-3384: - Labels: triaged (was: ) > COPY (IN)TO Missing docs > > > Key: ASTERIXDB-3384 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3384 > Project: Apache AsterixDB > Issue Type: Bug > Components: DOC - Documentation >Affects Versions: 0.9.9 >Reporter: Ian Maxon >Assignee: Ian Maxon >Priority: Major > Labels: triaged > Fix For: 0.9.9 > > > These two features don't have their accompanying SQL++ statements documented. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ASTERIXDB-3383) Add tuple filter to ingestion pipeline
[ https://issues.apache.org/jira/browse/ASTERIXDB-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Maxon updated ASTERIXDB-3383: - Labels: triaged (was: ) > Add tuple filter to ingestion pipeline > -- > > Key: ASTERIXDB-3383 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3383 > Project: Apache AsterixDB > Issue Type: Improvement > Components: RT - Runtime >Affects Versions: 0.9.10 >Reporter: Murtadha Makki Al Hubail >Assignee: Ritik Raj >Priority: Major > Labels: triaged > Fix For: 0.9.10 > > > Add a tuple filter that can be used to exclude tuple from being processed by > the ingestion pipeline. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ASTERIXDB-3382) Add Helm charts
[ https://issues.apache.org/jira/browse/ASTERIXDB-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Maxon updated ASTERIXDB-3382: - Labels: triaged (was: ) > Add Helm charts > --- > > Key: ASTERIXDB-3382 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3382 > Project: Apache AsterixDB > Issue Type: New Feature >Reporter: Ian Maxon >Assignee: Ian Maxon >Priority: Major > Labels: triaged > > Along with the containerized artifact, there should also be a simple Helm > chart that will deploy it to a kubernetes cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (ASTERIXDB-3380) Internal error on a syntax error with productivity hint
[
https://issues.apache.org/jira/browse/ASTERIXDB-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839059#comment-17839059
]
ASF subversion and git services commented on ASTERIXDB-3380:
Commit a19254aedf1c1b2d4e749d5f21e2f2e4984c37fc in asterixdb's branch
refs/heads/master from Vijay Sarathy
[ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=a19254aedf ]
[ASTERIXDB-3380][COMP] Internal error on a syntax error with productivity hint
Change-Id: I42399559414f94204b22897d0fae43ea8c0f54d4
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18238
Reviewed-by: Vijay Sarathy
Reviewed-by: Ali Alsuliman
Integration-Tests: Jenkins
Tested-by: Jenkins
> Internal error on a syntax error with productivity hint
> ---
>
> Key: ASTERIXDB-3380
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3380
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
>Reporter: murali krishna
>Assignee: Vijay Sarathy
>Priority: Major
> Labels: triaged
>
> {code:java}
> select count(1) from K1 x join K2 y on x.rand_i /*+ selectivity 0.065536 */
> <= 65536 and tobigint(x.u256_i) /*+ productivit y 3906.25 */ /*+ indexnl */ =
> y.u256_i;
> {code}
>
> productivity hint is misspelled. Hint should be ignored. Should not be an
> internal error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ASTERIXDB-3380) Internal error on a syntax error with productivity hint
[
https://issues.apache.org/jira/browse/ASTERIXDB-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Maxon updated ASTERIXDB-3380:
-
Labels: triaged (was: )
> Internal error on a syntax error with productivity hint
> ---
>
> Key: ASTERIXDB-3380
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3380
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
>Reporter: murali krishna
>Assignee: Vijay Sarathy
>Priority: Major
> Labels: triaged
>
> {code:java}
> select count(1) from K1 x join K2 y on x.rand_i /*+ selectivity 0.065536 */
> <= 65536 and tobigint(x.u256_i) /*+ productivit y 3906.25 */ /*+ indexnl */ =
> y.u256_i;
> {code}
>
> productivity hint is misspelled. Hint should be ignored. Should not be an
> internal error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ASTERIXDB-3381) Embed values when reading Avro files
[ https://issues.apache.org/jira/browse/ASTERIXDB-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Maxon updated ASTERIXDB-3381: - Labels: triaged (was: ) > Embed values when reading Avro files > > > Key: ASTERIXDB-3381 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3381 > Project: Apache AsterixDB > Issue Type: Improvement > Components: EXT - External data >Affects Versions: 0.9.10 >Reporter: Wail Y. Alkowaileet >Assignee: Wail Y. Alkowaileet >Priority: Major > Labels: triaged > Fix For: 0.9.10 > > > Add the support for filter value embedding when parsing Avro files -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (ASTERIXDB-3378) Allow user to opt out of validating write permission before COPY TO S3
[
https://issues.apache.org/jira/browse/ASTERIXDB-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Maxon resolved ASTERIXDB-3378.
--
Resolution: Fixed
> Allow user to opt out of validating write permission before COPY TO S3
> --
>
> Key: ASTERIXDB-3378
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3378
> Project: Apache AsterixDB
> Issue Type: Task
> Components: EXT - External data
>Affects Versions: 0.9.9
>Reporter: Hussain Towaileb
>Assignee: Hussain Towaileb
>Priority: Major
> Fix For: 0.9.9
>
>
> Currently, for COPY TO S3, one of our early checks validates that:
> * We have the write permission, by writing a small test file to the
> destination, then deleting it, which also tests that
> * Multipart upload is supported, as this is the form of operation we use for
> writing.
>
> This test helps identify failure scenarios early before processing (for
> example) large amount of data, only to fail eventually.
> The limitation of this validation is that it is not always possible to know
> the final destination we will be writing to, depending on the statement
> itself. And so, we always perform this test at the root of the bucket.
> According to the S3 docs, it seems it might be possible to grant the
> permissions only on specific prefixes in a bucket, but not all. This can be
> problematic if we have permission on our destination prefix, but not the root
> of the bucket, because the validation will keep trying to validate against
> the root, which will lead to a failure of the operation.
> This ticket suggests the addition of a flag, e.g.
> {*}"validate-write-permission" (default to true){*}, to skip this operation
> and let the user carry the responsibility of having the right permissions at
> the right time. Note this will only skip the write permission validation, all
> other validations will still take place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ASTERIXDB-3378) Allow user to opt out of validating write permission before COPY TO S3
[
https://issues.apache.org/jira/browse/ASTERIXDB-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Maxon updated ASTERIXDB-3378:
-
Labels: triaged (was: )
> Allow user to opt out of validating write permission before COPY TO S3
> --
>
> Key: ASTERIXDB-3378
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3378
> Project: Apache AsterixDB
> Issue Type: Task
> Components: EXT - External data
>Affects Versions: 0.9.9
>Reporter: Hussain Towaileb
>Assignee: Hussain Towaileb
>Priority: Major
> Labels: triaged
> Fix For: 0.9.9
>
>
> Currently, for COPY TO S3, one of our early checks validates that:
> * We have the write permission, by writing a small test file to the
> destination, then deleting it, which also tests that
> * Multipart upload is supported, as this is the form of operation we use for
> writing.
>
> This test helps identify failure scenarios early before processing (for
> example) large amount of data, only to fail eventually.
> The limitation of this validation is that it is not always possible to know
> the final destination we will be writing to, depending on the statement
> itself. And so, we always perform this test at the root of the bucket.
> According to the S3 docs, it seems it might be possible to grant the
> permissions only on specific prefixes in a bucket, but not all. This can be
> problematic if we have permission on our destination prefix, but not the root
> of the bucket, because the validation will keep trying to validate against
> the root, which will lead to a failure of the operation.
> This ticket suggests the addition of a flag, e.g.
> {*}"validate-write-permission" (default to true){*}, to skip this operation
> and let the user carry the responsibility of having the right permissions at
> the right time. Note this will only skip the write permission validation, all
> other validations will still take place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (ASTERIXDB-3379) Provide Azure Blob Storage support for the Engine
[ https://issues.apache.org/jira/browse/ASTERIXDB-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Maxon updated ASTERIXDB-3379: - Labels: triaged (was: ) > Provide Azure Blob Storage support for the Engine > - > > Key: ASTERIXDB-3379 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-3379 > Project: Apache AsterixDB > Issue Type: New Feature > Components: STO - Storage >Affects Versions: 0.9.10 >Reporter: Mohammad Nawazish Khan >Priority: Major > Labels: triaged > Fix For: 0.9.9 > > > In order to provide compute/storage separation, the engine is to support > reading and writing it's data from Azure Blob Storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
>From Wail Alkowaileet :
Attention is currently required from: Peeyush Gupta, Ian Maxon, Hussain
Towaileb.
Wail Alkowaileet has posted comments on this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 )
Change subject: [ASTERIXDB-3384][DOC] Document COPY
..
Patch Set 1:
(3 comments)
File asterixdb/asterix-doc/src/main/markdown/sqlpp/7_ddl_dml.md:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243/comment/9eefca59_990991d2
PS1, Line 734: localfs
Probably it is better to keep everything as S3. Not that won't work, but it is
going to be weird reading those files in a distributed setting.
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243/comment/def178a6_4261270e
PS1, Line 752: PATH("CopyToResult/" || to_string(b))
PATH("CopyToResult", to_string(b)) is much cleaner
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243/comment/e3611820_6846466b
PS1, Line 760: 100
I believe this will throw an error. The minimum is 1000
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c
Gerrit-Change-Number: 18243
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Hussain Towaileb
Gerrit-Reviewer: Ian Maxon
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Peeyush Gupta
Gerrit-CC: Wail Alkowaileet
Gerrit-Attention: Peeyush Gupta
Gerrit-Attention: Ian Maxon
Gerrit-Attention: Hussain Towaileb
Gerrit-Comment-Date: Fri, 19 Apr 2024 16:57:19 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
>From Jenkins : Attention is currently required from: Peeyush Gupta, Hussain Towaileb. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 ) Change subject: [ASTERIXDB-3384][DOC] Document COPY .. Patch Set 1: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/328/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c Gerrit-Change-Number: 18243 Gerrit-PatchSet: 1 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Fri, 19 Apr 2024 16:55:58 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: WIP: Duplicate fields accepted as primary key
>From Jenkins : Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 ) Change subject: WIP: Duplicate fields accepted as primary key .. Patch Set 1: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/327/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28 Gerrit-Change-Number: 18244 Gerrit-PatchSet: 1 Gerrit-Owner: Peeyush Gupta Gerrit-Reviewer: Jenkins Gerrit-Comment-Date: Fri, 19 Apr 2024 16:40:43 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin
>From Ian Maxon : Attention is currently required from: Michael Blow. Ian Maxon has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 ) Change subject: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin .. Patch Set 8: Code-Review+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b4420a4102c122f8af1d Gerrit-Change-Number: 18183 Gerrit-PatchSet: 8 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Fri, 19 Apr 2024 16:20:43 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
>From Ian Maxon : Attention is currently required from: Peeyush Gupta, Hussain Towaileb. Ian Maxon has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 ) Change subject: [ASTERIXDB-3384][DOC] Document COPY .. Patch Set 1: Code-Review+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c Gerrit-Change-Number: 18243 Gerrit-PatchSet: 1 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Peeyush Gupta Gerrit-Attention: Peeyush Gupta Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Fri, 19 Apr 2024 16:20:19 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
[jira] [Resolved] (ASTERIXDB-3380) Internal error on a syntax error with productivity hint
[
https://issues.apache.org/jira/browse/ASTERIXDB-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Sarathy resolved ASTERIXDB-3380.
--
Resolution: Fixed
> Internal error on a syntax error with productivity hint
> ---
>
> Key: ASTERIXDB-3380
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3380
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
>Reporter: murali krishna
>Assignee: Vijay Sarathy
>Priority: Major
>
> {code:java}
> select count(1) from K1 x join K2 y on x.rand_i /*+ selectivity 0.065536 */
> <= 65536 and tobigint(x.u256_i) /*+ productivit y 3906.25 */ /*+ indexnl */ =
> y.u256_i;
> {code}
>
> productivity hint is misspelled. Hint should be ignored. Should not be an
> internal error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3380][COMP] Internal error on a syntax error with producti...
>From Vijay Sarathy : Vijay Sarathy has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18238 ) Change subject: [ASTERIXDB-3380][COMP] Internal error on a syntax error with productivity hint .. [ASTERIXDB-3380][COMP] Internal error on a syntax error with productivity hint Change-Id: I42399559414f94204b22897d0fae43ea8c0f54d4 Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18238 Reviewed-by: Vijay Sarathy Reviewed-by: Ali Alsuliman Integration-Tests: Jenkins Tested-by: Jenkins --- A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.06.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.07.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.08.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/warnings/multiple-hints-warning/multiple-hints-warning.06.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/warnings/multiple-hints-warning/multiple-hints-warning.08.adm A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.05.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml A asterixdb/asterix-app/src/test/resources/runtimets/results/warnings/multiple-hints-warning/multiple-hints-warning.05.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/warnings/multiple-hints-warning/multiple-hints-warning.07.adm M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.09.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/warnings/multiple-hints-warning/multiple-hints-warning.09.adm 12 files changed, 210 insertions(+), 13 deletions(-) Approvals: Ali Alsuliman: Looks good to me, approved Vijay Sarathy: Looks good to me, but someone else must approve Jenkins: Verified; Verified Objections: Anon. E. Moose #1000171: Violations found diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.05.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.05.query.sqlpp new file mode 100644 index 000..64d8d90 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.05.query.sqlpp @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/* +* Description : Test warnings for multiple hints +* Expected Res : Warning, ignore hint +* Date : 11/20/2022 +*/ +// requesttype=application/json +// param max-warnings:json=10 + +use tpch; + +SELECT COUNT(*) FROM customer c JOIN nation n +ON c.c_nationkey /*+ hashjoin build (n) */ /*+ productivit y 100.0 */ = n.n_nationkey; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.06.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.06.query.sqlpp new file mode 100644 index 000..2cdd902 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/warnings/multiple-hints-warning/multiple-hints-warning.06.query.sqlpp @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by
Change in asterixdb[master]: [ASTERIXDB-3380][COMP] Internal error on a syntax error with producti...
>From Ali Alsuliman : Attention is currently required from: Vijay Sarathy. Ali Alsuliman has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18238 ) Change subject: [ASTERIXDB-3380][COMP] Internal error on a syntax error with productivity hint .. Patch Set 3: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18238 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I42399559414f94204b22897d0fae43ea8c0f54d4 Gerrit-Change-Number: 18238 Gerrit-PatchSet: 3 Gerrit-Owner: Vijay Sarathy Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Vijay Sarathy Gerrit-Attention: Vijay Sarathy Gerrit-Comment-Date: Fri, 19 Apr 2024 16:08:28 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: WIP: Duplicate fields accepted as primary key
>From Peeyush Gupta :
Peeyush Gupta has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18244 )
Change subject: WIP: Duplicate fields accepted as primary key
..
WIP: Duplicate fields accepted as primary key
Change-Id: I07f3bca08058dd17fd54dd0e9cc6b4ee8cb01d28
---
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-app/src/test/resources/runtimets/sqlpp_queries.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.04.ddl.sqlpp
M
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
6 files changed, 89 insertions(+), 0 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/44/18244/1
diff --git
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
index e253e07..4d854c1 100644
---
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
+++
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
@@ -122,10 +122,36 @@
autogenerated, true, sourceLoc, partitioningExprTypes);
}
+private static void checkDuplicateFields(List>
partitioningExprs, SourceLocation sourceLoc)
+throws CompilationException {
+for (int i = 0; i < partitioningExprs.size(); i++) {
+for (int j = i + 1; j < partitioningExprs.size(); j++) {
+int k = 0;
+List expr1 = partitioningExprs.get(i);
+List expr2 = partitioningExprs.get(j);
+while (k < expr1.size() && k < expr2.size()) {
+if (!expr1.get(k).equals(expr2.get(k))) {
+break;
+} else {
+k += 1;
+}
+}
+if (k == expr1.size() && k == expr2.size()) {
+throw new
CompilationException(ErrorCode.DUPLICATE_FIELD_IN_PRIMARY_KEY, sourceLoc,
+String.join(".", expr1));
+} else if (k == expr1.size() || k == expr2.size()) {
+throw new
CompilationException(ErrorCode.INCOMPATIBLE_FIELDS_IN_PRIMARY_KEY, sourceLoc,
+String.join(".", expr1), String.join(".", expr2));
+}
+}
+}
+}
+
private static List
validatePartitioningExpressionsImpl(ARecordType recType, ARecordType
metaRecType,
List> partitioningExprs, List
keySourceIndicators, boolean autogenerated,
boolean forPrimaryKey, SourceLocation sourceLoc,
List partitioningExprTypes)
throws AlgebricksException {
+checkDuplicateFields(partitioningExprs, sourceLoc);
String keyKindDisplayName = forPrimaryKey ? PRIMARY : "";
List computedPartitioningExprTypes = new
ArrayList<>(partitioningExprs.size());
if (autogenerated) {
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
new file mode 100644
index 000..52c5119
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ddl/create-dataset-negative/create-dataset-negative.03.ddl.sqlpp
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+drop dataverse test if exists;
+create dataverse test;
+use test;
+
+create dataset orders primary key (id:int, id:int);
diff --git
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
Attention is currently required from: Ian Maxon. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 ) Change subject: [ASTERIXDB-3384][DOC] Document COPY .. Patch Set 1: Contrib+1 Analytics Compatibility Tests Successful https://cbjenkins.page.link/p156rtFtA1gCwL5B6 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c Gerrit-Change-Number: 18243 Gerrit-PatchSet: 1 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Ian Maxon Gerrit-Comment-Date: Fri, 19 Apr 2024 12:04:54 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
>From Jenkins : Attention is currently required from: Ian Maxon. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 ) Change subject: [ASTERIXDB-3384][DOC] Document COPY .. Patch Set 1: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/326/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c Gerrit-Change-Number: 18243 Gerrit-PatchSet: 1 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Attention: Ian Maxon Gerrit-Comment-Date: Fri, 19 Apr 2024 10:17:34 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 ) Change subject: [ASTERIXDB-3384][DOC] Document COPY .. Patch Set 1: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/2ttifn43HhfZKEMc7 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c Gerrit-Change-Number: 18243 Gerrit-PatchSet: 1 Gerrit-Owner: Ian Maxon Gerrit-CC: Anon. E. Moose #1000171 Gerrit-CC: Jenkins Gerrit-Comment-Date: Fri, 19 Apr 2024 09:50:19 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
[jira] [Created] (ASTERIXDB-3384) COPY (IN)TO Missing docs
Ian Maxon created ASTERIXDB-3384: Summary: COPY (IN)TO Missing docs Key: ASTERIXDB-3384 URL: https://issues.apache.org/jira/browse/ASTERIXDB-3384 Project: Apache AsterixDB Issue Type: Bug Components: DOC - Documentation Affects Versions: 0.9.9 Reporter: Ian Maxon Assignee: Ian Maxon Fix For: 0.9.9 These two features don't have their accompanying SQL++ statements documented. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3384][DOC] Document COPY
>From Ian Maxon :
Ian Maxon has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243 )
Change subject: [ASTERIXDB-3384][DOC] Document COPY
..
[ASTERIXDB-3384][DOC] Document COPY
Details:
Add some brief documentation about each COPY
statement. Also add the simplified BNF for railroad
diagrams for each.
Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c
---
M asterixdb/asterix-doc/src/main/grammar/sqlpp.ebnf
M asterixdb/asterix-doc/src/main/markdown/sqlpp/7_ddl_dml.md
2 files changed, 88 insertions(+), 0 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/43/18243/1
diff --git a/asterixdb/asterix-doc/src/main/grammar/sqlpp.ebnf
b/asterixdb/asterix-doc/src/main/grammar/sqlpp.ebnf
index af67e33..31df730 100644
--- a/asterixdb/asterix-doc/src/main/grammar/sqlpp.ebnf
+++ b/asterixdb/asterix-doc/src/main/grammar/sqlpp.ebnf
@@ -258,4 +258,8 @@
UpsertStmnt ::= "UPSERT" "INTO" QualifiedName ("AS" Variable)? Query
("RETURNING" Expr)?
+CopyStmnt ::= "COPY" "INTO"? QualifiedName ("AS" Variable)? "FROM" Identifier
"AT" QualifiedName ("PATH" StringLiteral)? (WITH ObjectConstructor)?
+
+CopyToStmnt ::= "COPY" ( QualifiedName | "(" Query ")" ) "TO" AdapterName
"PATH" ParenthesizedArrayConstructor ("OVER" "(" ("PARTITION" "BY" Expr ("AS"
Variable)? ("," Expr ("AS" Variable)? )? )? OrderbyClause ")" )?
+
DeleteStmnt ::= "DELETE" "FROM" QualifiedName (("AS")? Variable)? ("WHERE"
Expr)?
diff --git a/asterixdb/asterix-doc/src/main/markdown/sqlpp/7_ddl_dml.md
b/asterixdb/asterix-doc/src/main/markdown/sqlpp/7_ddl_dml.md
index 4ec5033..a7a 100644
--- a/asterixdb/asterix-doc/src/main/markdown/sqlpp/7_ddl_dml.md
+++ b/asterixdb/asterix-doc/src/main/markdown/sqlpp/7_ddl_dml.md
@@ -697,3 +697,73 @@
DELETE FROM customers WHERE custid = "C47";
+### Copy Statement
+
+# CopyStmnt
+
+
+The `COPY` statement is used to load data in bulk from an external source into
a dataset. It differs from `LOAD` in that
+it can be performed multiple times, to upsert new data into the dataset that
may have changed in the source.
+
+For example, this statement would copy the contents of the JSON file
`customerData.json` residing on the NC named `asterix_nc1`
+into the Customers dataset.
+
+# Example
+
+COPY Customers
+FROM localfs
+PATH ("asterix_nc1://data/nontagged/customerData.json")
+WITH {
+"format": "json"
+};
+
+
+### Copy To Statement
+
+# CopyToStmnt
+
+
+The `COPY TO` statement allows easy export of the result of a query, or an
entire dataset, into a file or set of files on
+an external source. This can be any source that has an adapter.
+
+For example, this statement would create a copy of `ColumnDataset` on each
node as a single JSON file
+
+# Example
+
+COPY ColumnDataset
+TO localfs
+PATH("localhost:///media/backup/CopyToResult")
+WITH {
+"format" : "json"
+};
+
+The statement allows for much more precise exports than this, however. A
typical pattern of data accessed via object stores
+like S3 is for it to be partitioned into files, with each folder containing
some of those files representing a key. The use
+of `OVER` and `PARTITION BY` allow exports to match this. For example:
+
+# Example
+
+COPY (SELECT cd.uid uid,
+cd.sensor_info.name name,
+to_bigint(cd.sensor_info.battery_status) battery_status
+FROM ColumnDataset cd
+) toWrite
+TO s3
+PATH("CopyToResult/" || to_string(b))
+OVER (
+PARTITION BY toWrite.battery_status b
+ORDER BY toWrite.name
+)
+WITH {
+"format" : "json",
+"compression": "gzip",
+"max-objects-per-file": 100,
+"container": "myBucket",
+"accessKeyId": "",
+"secretAccessKey": "",
+"region": "us-west-2"
+};
+
+This query will be exported as partitions into a set of folders, with one
folder for each value of `battery_status`.
+Each partition itself will also be sorted by the `name` field, and compressed
with `gzip` and divided into files of 100
+objects or fewer per file.
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18243
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b1420a4102c122f8bf1c
Gerrit-Change-Number: 18243
Gerrit-PatchSet: 1
Gerrit-Owner: Ian Maxon
Gerrit-MessageType: newchange
[jira] [Commented] (ASTERIXDB-3378) Allow user to opt out of validating write permission before COPY TO S3
[
https://issues.apache.org/jira/browse/ASTERIXDB-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838870#comment-17838870
]
ASF subversion and git services commented on ASTERIXDB-3378:
Commit ab0abc111eaa663f9115756ba4e7cbf1da1c5f3b in asterixdb's branch
refs/heads/master from Hussain Towaileb
[ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=ab0abc111e ]
[ASTERIXDB-3378][EXT]: Allow user to opt out of validating write permission
before COPY TO S3
Change-Id: Ie099710714e032674727d0cb426cc450d8dda167
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234
Integration-Tests: Jenkins
Tested-by: Jenkins
Reviewed-by: Hussain Towaileb
> Allow user to opt out of validating write permission before COPY TO S3
> --
>
> Key: ASTERIXDB-3378
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-3378
> Project: Apache AsterixDB
> Issue Type: Task
> Components: EXT - External data
>Affects Versions: 0.9.9
>Reporter: Hussain Towaileb
>Assignee: Hussain Towaileb
>Priority: Major
> Fix For: 0.9.9
>
>
> Currently, for COPY TO S3, one of our early checks validates that:
> * We have the write permission, by writing a small test file to the
> destination, then deleting it, which also tests that
> * Multipart upload is supported, as this is the form of operation we use for
> writing.
>
> This test helps identify failure scenarios early before processing (for
> example) large amount of data, only to fail eventually.
> The limitation of this validation is that it is not always possible to know
> the final destination we will be writing to, depending on the statement
> itself. And so, we always perform this test at the root of the bucket.
> According to the S3 docs, it seems it might be possible to grant the
> permissions only on specific prefixes in a bucket, but not all. This can be
> problematic if we have permission on our destination prefix, but not the root
> of the bucket, because the validation will keep trying to validate against
> the root, which will lead to a failure of the operation.
> This ticket suggests the addition of a flag, e.g.
> {*}"validate-write-permission" (default to true){*}, to skip this operation
> and let the user carry the responsibility of having the right permissions at
> the right time. Note this will only skip the write permission validation, all
> other validations will still take place.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Change in asterixdb[master]: [ASTERIXDB-3378][EXT]: Allow user to opt out of validating write perm...
>From Hussain Towaileb :
Hussain Towaileb has submitted this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234 )
Change subject: [ASTERIXDB-3378][EXT]: Allow user to opt out of validating
write permission before COPY TO S3
..
[ASTERIXDB-3378][EXT]: Allow user to opt out of validating write permission
before COPY TO S3
Change-Id: Ie099710714e032674727d0cb426cc450d8dda167
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234
Integration-Tests: Jenkins
Tested-by: Jenkins
Reviewed-by: Hussain Towaileb
---
M
asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/writer/S3ExternalFileWriterFactory.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
2 files changed, 21 insertions(+), 0 deletions(-)
Approvals:
Hussain Towaileb: Looks good to me, approved
Jenkins: Verified; Verified
Anon. E. Moose #1000171:
diff --git
a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/writer/S3ExternalFileWriterFactory.java
b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/writer/S3ExternalFileWriterFactory.java
index 5faa8e1..cdaa6dc 100644
---
a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/writer/S3ExternalFileWriterFactory.java
+++
b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/writer/S3ExternalFileWriterFactory.java
@@ -151,6 +151,13 @@
}
}
+// do not validate write permissions if specified by the user not to
do so
+String validateWritePermissions = configuration
+
.getOrDefault(ExternalDataConstants.KEY_VALIDATE_WRITE_PERMISSION,
Boolean.TRUE.toString());
+if (!Boolean.parseBoolean(validateWritePermissions)) {
+return;
+}
+
Random random = new Random();
String pathPrefix = "testFile";
String path = pathPrefix + random.nextInt();
diff --git
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
index fc8..98ccbce 100644
---
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
+++
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
@@ -310,6 +310,7 @@
* Writer Constants
*/
public static final String KEY_WRITER_MAX_RESULT = "max-objects-per-file";
+public static final String KEY_VALIDATE_WRITE_PERMISSION =
"validate-write-permission";
public static final String KEY_WRITER_COMPRESSION = "compression";
public static final int WRITER_MAX_RESULT_DEFAULT = 1;
public static final int WRITER_MAX_RESULT_MINIMUM = 1000;
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Ie099710714e032674727d0cb426cc450d8dda167
Gerrit-Change-Number: 18234
Gerrit-PatchSet: 4
Gerrit-Owner: Hussain Towaileb
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Hussain Towaileb
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Wail Alkowaileet
Gerrit-MessageType: merged
Change in asterixdb[master]: [ASTERIXDB-3378][EXT]: Allow user to opt out of validating write perm...
>From Hussain Towaileb : Attention is currently required from: Wail Alkowaileet. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234 ) Change subject: [ASTERIXDB-3378][EXT]: Allow user to opt out of validating write permission before COPY TO S3 .. Patch Set 3: Code-Review+2 (3 comments) Patchset: PS3: Forwardning Wail's +2 File asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/writer/S3ExternalFileWriterFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234/comment/1a0934c6_0d199940 PS2, Line 155: get > getOrDefault? Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234/comment/492abd17_c18380ce PS2, Line 156: Boolean.parseBoolean(validateWritePermissions) > This returns boolean (not Boolean). It is fine to do !Boolean. […] Done -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18234 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ie099710714e032674727d0cb426cc450d8dda167 Gerrit-Change-Number: 18234 Gerrit-PatchSet: 3 Gerrit-Owner: Hussain Towaileb Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Wail Alkowaileet Gerrit-Comment-Date: Fri, 19 Apr 2024 06:11:03 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Comment-In-Reply-To: Wail Alkowaileet Gerrit-MessageType: comment
