[jira] [Resolved] (ASTERIXDB-2599) Look Aside Files (LAF) are not cleaned up after merge
[ https://issues.apache.org/jira/browse/ASTERIXDB-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet resolved ASTERIXDB-2599. - Resolution: Fixed > Look Aside Files (LAF) are not cleaned up after merge > - > > Key: ASTERIXDB-2599 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2599 > Project: Apache AsterixDB > Issue Type: Bug > Components: STO - Storage >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > > LAF files are not cleaned up after merge operations. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (ASTERIXDB-2599) Look Aside Files (LAF) are not cleaned up after merge
[ https://issues.apache.org/jira/browse/ASTERIXDB-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet reassigned ASTERIXDB-2599: --- Assignee: Wail Alkowaileet > Look Aside Files (LAF) are not cleaned up after merge > - > > Key: ASTERIXDB-2599 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2599 > Project: Apache AsterixDB > Issue Type: Bug > Components: STO - Storage >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > > LAF files are not cleaned up after merge operations. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (ASTERIXDB-2599) Look Aside Files (LAF) are not cleaned up after merge
Wail Alkowaileet created ASTERIXDB-2599: --- Summary: Look Aside Files (LAF) are not cleaned up after merge Key: ASTERIXDB-2599 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2599 Project: Apache AsterixDB Issue Type: Bug Components: STO - Storage Reporter: Wail Alkowaileet LAF files are not cleaned up after merge operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2517) Ingestion process failed on a cluster with two machines.
[ https://issues.apache.org/jira/browse/ASTERIXDB-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830701#comment-16830701 ] Wail Alkowaileet commented on ASTERIXDB-2517: - Ok .. my machine was under attack. Someone was trying to gain access using SSH: Apr 30 12:44:00 wail-desktop sshd[18273]: Failed password for root from 218.92.0.179 port 24645 ssh2 Apr 30 12:44:02 wail-desktop sshd[18275]: Failed password for root from 218.92.0.141 port 17552 ssh2 Apr 30 12:44:02 wail-desktop sshd[18277]: Failed password for root from 218.92.0.141 port 18157 ssh2 Apr 30 12:44:03 wail-desktop sshd[18273]: Failed password for root from 218.92.0.179 port 24645 ssh2 Apr 30 12:44:04 wail-desktop sshd[18275]: Failed password for root from 218.92.0.141 port 17552 ssh2 Apr 30 12:44:04 wail-desktop sshd[18277]: Failed password for root from 218.92.0.141 port 18157 ssh2 Apr 30 12:44:06 wail-desktop sshd[18273]: Failed password for root from 218.92.0.179 port 24645 ssh2 Apr 30 12:44:07 wail-desktop sshd[18275]: Failed password for root from 218.92.0.141 port 17552 ssh2 Apr 30 12:44:07 wail-desktop sshd[18277]: Failed password for root from 218.92.0.141 port 18157 ssh2 Apr 30 12:44:08 wail-desktop sshd[18273]: Failed password for root from 218.92.0.179 port 24645 ssh2 I changed my port and it seems the problem is gone. I was getting the error so frequently, almost every 20 minutes. > Ingestion process failed on a cluster with two machines. > > > Key: ASTERIXDB-2517 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2517 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Priority: Major > Attachments: cc.log, nc-1.log > > > We have a cluster with two machines. Out of 1.5 billion records, about 1.2 > billion records were ingested using a socket adapter. However, the NC-1, > which is located on the same machine, was shutdown. The time was around 19:21 > (please see the log records around that time). I have attached the log > records. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2517) Ingestion process failed on a cluster with two machines.
[
https://issues.apache.org/jira/browse/ASTERIXDB-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830654#comment-16830654
]
Wail Alkowaileet commented on ASTERIXDB-2517:
-
I got the same issue as well. Except there is no ingestion in my case. I have
similar configuration (one cc and one nc) in the same machine.
{code:java}
12:45:14.422 [TCPEndpoint IO Thread [null]] ERROR
org.apache.hyracks.net.protocols.tcp.TCPEndpoint - Unexpected tcp io error in
connection TCPConnection[Remote Address: /127.0.0.1:36955 Local Address: null]
org.apache.hyracks.api.exceptions.NetException: Socket Closed
at
org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.driveReaderStateMachine(MultiplexedConnection.java:342)
~[hyracks-net-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.notifyIOReady(MultiplexedConnection.java:113)
~[hyracks-net-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:186)
[hyracks-net-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
12:46:04.244 [Executor-3:ClusterController] INFO
org.apache.hyracks.control.cc.cluster.NodeManager - Requesting node 1 to
shutdown to ensure failure
12:46:04.245 [Worker:ClusterController] INFO
org.apache.hyracks.control.cc.cluster.NodeManager - 1 considered dead. Last
heartbeat received 50558ms ago. Max miss period: 5ms
12:46:04.245 [Worker:ClusterController] INFO
org.apache.hyracks.control.cc.work.RemoveDeadNodesWork - Number of affected
jobs: 1
12:46:04.249 [Executor-3:ClusterController] WARN
org.apache.hyracks.ipc.impl.ReconnectingIPCHandle - ipcHandle IPCHandle
[addr=/127.0.0.1:44551 state=CLOSED] disconnected; will attempt to reconnect 1
times
12:46:04.251 [IPC Network Listener Thread [/0:0:0:0:0:0:0:0:1099]] WARN
org.apache.hyracks.ipc.impl.IPCConnectionManager - Exception finishing connect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_201]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
~[?:1.8.0_201]
at
org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.finishConnect(IPCConnectionManager.java:239)
[hyracks-ipc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.processSelectedKeys(IPCConnectionManager.java:229)
[hyracks-ipc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:200)
[hyracks-ipc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)
[hyracks-ipc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
12:46:04.256 [IPC Network Listener Thread [/0:0:0:0:0:0:0:0:1099]] WARN
org.apache.hyracks.ipc.impl.IPCConnectionManager - Failed to finish connect to
/127.0.0.1:44551
12:46:04.257 [Executor-3:ClusterController] WARN
org.apache.hyracks.ipc.impl.IPCConnectionManager - Connection to
/127.0.0.1:44551 failed; retrying (retry attempt 1 of 1) after 100ms
12:46:04.265 [Worker:ClusterController] ERROR
org.apache.hyracks.control.cc.executor.JobExecutor - Unexpected failure.
Aborting job JID:0.13
org.apache.hyracks.api.exceptions.HyracksException: HYR0010: Node 1 does not
exist
at
org.apache.hyracks.api.exceptions.HyracksException.create(HyracksException.java:57)
~[hyracks-api-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.cc.executor.JobExecutor.assignLocation(JobExecutor.java:473)
~[hyracks-control-cc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.cc.executor.JobExecutor.assignTaskLocations(JobExecutor.java:365)
~[hyracks-control-cc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.cc.executor.JobExecutor.startRunnableTaskClusters(JobExecutor.java:245)
~[hyracks-control-cc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.cc.executor.JobExecutor.startRunnableActivityClusters(JobExecutor.java:209)
~[hyracks-control-cc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.cc.executor.JobExecutor.notifyNodeFailures(JobExecutor.java:732)
[hyracks-control-cc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.cc.work.RemoveDeadNodesWork.run(RemoveDeadNodesWork.java:60)
[hyracks-control-cc-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
at
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:127)
[hyracks-control-common-0.3.5-SNAPSHOT.jar:0.3.5-SNAPSHOT]
12:46:04.265 [Worker:ClusterController] INFO
org.apache.asterix.hyracks.bootstrap.ClusterLifecycleListener - NC: 1 left
12:46:04.265 [Worker:ClusterController] INFO
org.apache.asterix.runtime.utils.ClusterStateManager - Removing configuration
parameters for node id 1
12:46:04.265 [Worker:ClusterController] INFO
org.apache.asterix.runtime.utils.ClusterStateManager -
[jira] [Resolved] (ASTERIXDB-2480) JSON Parser failed at NPE
[
https://issues.apache.org/jira/browse/ASTERIXDB-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet resolved ASTERIXDB-2480.
-
Resolution: Fixed
Fix Version/s: 0.9.5
> JSON Parser failed at NPE
> -
>
> Key: ASTERIXDB-2480
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2480
> Project: Apache AsterixDB
> Issue Type: Bug
>Affects Versions: 0.9.4
>Reporter: Xikui Wang
>Assignee: Wail Alkowaileet
>Priority: Major
> Fix For: 0.9.5
>
>
> {code}
> drop dataverse lapd if exists;
> create dataverse lapd;
> use lapd;
> create type WorrisomeNotificationType as {
> dataverseName: string,
> channelName: string
> };
> create feed WorrisomeTweetsFeed with {
> "adapter-name" : "http_adapter",
> "addresses" : "127.0.0.1:10011",
> "address-type" : "IP",
> "type-name" : "WorrisomeNotificationType",
> "format" : "json"
> };
> create dataset WorrisomeNotifications(WorrisomeNotificationType) primary key
> dataverseName;
> connect feed WorrisomeTweetsFeed to dataset WorrisomeNotifications;
> start feed WorrisomeTweetsFeed;
> {code}
> feed following json data, a NPE will be thrown.
> { "dataverseName":"dhs", "channelName":"WorrisomeTweetsNearMonumentsChannel",
> "channelExecutionTime":"2018-11-12T18:43:48.508Z"}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ASTERIXDB-2492) Skip filter for undo
[ https://issues.apache.org/jira/browse/ASTERIXDB-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet resolved ASTERIXDB-2492. - Resolution: Fixed Fix Version/s: 0.9.5 > Skip filter for undo > > > Key: ASTERIXDB-2492 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2492 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Affects Versions: 0.9.5 >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > Fix For: 0.9.5 > > Attachments: nc-asterix_nc1-12-07-18.log.gz > > > Currently, the RecoveryManager is not skipping the filter for undo. This > means currently the filter tuple is storing arbitrary value when undo > performed (for instance inserting a tuple with duplicate key). > This bug appeared after the change at > [ASTERIXDB-2491|https://issues.apache.org/jira/browse/ASTERIXDB-2491] > Attached log. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2531) Normalized keys are ignored for optional fields
Wail Alkowaileet created ASTERIXDB-2531:
---
Summary: Normalized keys are ignored for optional fields
Key: ASTERIXDB-2531
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2531
Project: Apache AsterixDB
Issue Type: Improvement
Reporter: Wail Alkowaileet
Normalized keys are not utilized when fields are optional.
{code:java}
SELECT uname, a
FROM Tweets t
GROUP BY t.user.name AS uname WITH a AS AVG(LENGTH(text))
ORDER BY a DESC
LIMIT 10
{code}
_AVG(LENGTH(text))_ should always return NULL/MISSING or DOUBLE (despite if
_text_ is open or closed). It fails if text is not a string.
The issue is the *INormalizedKeyComputerFactory* does not know how to deal with
MISSING/NULL. Supporting the normalized keys for sorting can improve the
sorting time.
We might need to extend SQL++ to specify how to sort UNKNOWABLES:
{code:java}
SELECT uname, a
FROM Tweets t
GROUP BY t.user.name AS uname WITH a AS AVG(LENGTH(text))
ORDER BY a DESC (UNKNOWABLES (FIRST | LAST))?
LIMIT 10
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ASTERIXDB-1726) Recovery fails because of negative value
[
https://issues.apache.org/jira/browse/ASTERIXDB-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet resolved ASTERIXDB-1726.
-
Resolution: Fixed
> Recovery fails because of negative value
>
>
> Key: ASTERIXDB-1726
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1726
> Project: Apache AsterixDB
> Issue Type: Bug
>Reporter: Yingyi Bu
>Assignee: Ian Maxon
>Priority: Critical
>
> {noformat}
> org.apache.hyracks.storage.am.btree.exceptions.BTreeException:
> java.lang.IllegalArgumentException: The length must be an non-negative value
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:827)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.insertUpdateOrDelete(BTree.java:312)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.upsert(BTree.java:345)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.access$500(BTree.java:74)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.upsertIfConditionElseInsert(BTree.java:953)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.insert(LSMBTree.java:402)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.modify(LSMBTree.java:343)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:376)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.forceModify(LSMHarness.java:356)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.forceInsert(LSMTreeIndexAccessor.java:159)
> at
> org.apache.asterix.transaction.management.service.recovery.RecoveryManager.redo(RecoveryManager.java:884)
> at
> org.apache.asterix.transaction.management.service.recovery.RecoveryManager.startRecoveryRedoPhase(RecoveryManager.java:397)
> at
> org.apache.asterix.transaction.management.service.recovery.RecoveryManager.replayPartitionsLogs(RecoveryManager.java:202)
> at
> org.apache.asterix.transaction.management.service.recovery.RecoveryManager.startRecovery(RecoveryManager.java:194)
> at
> org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint.start(NCApplicationEntryPoint.java:151)
> at
> org.apache.hyracks.control.nc.NodeControllerService.startApplication(NodeControllerService.java:325)
> at
> org.apache.hyracks.control.nc.NodeControllerService.start(NodeControllerService.java:257)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2491) Recovery fail for large tuples (short integer overflow)
[ https://issues.apache.org/jira/browse/ASTERIXDB-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787033#comment-16787033 ] Wail Alkowaileet commented on ASTERIXDB-2491: - Thanks Mike B. The bug has been reported multiple times.. I marked them as duplicate. > Recovery fail for large tuples (short integer overflow) > --- > > Key: ASTERIXDB-2491 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2491 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Critical > Fix For: 0.9.5 > > Attachments: cc.log, nc-1.log > > > I've been running some tests with pretty large objects (~100KB). But it seems > the field end offsets in the log buffer are of type short [1, 2]. For large > fields (> 32KB), the field end offset is going to be a negative value due to > overflow. When recovery kicks in, I get an exception that "The length must be > a non-negative value". > [1] > [SimpleTupleWriter|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java#L106] > [2] > [SimpleTupleReference|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java#L86] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ASTERIXDB-2501) Redo failed at negative length value
[ https://issues.apache.org/jira/browse/ASTERIXDB-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet resolved ASTERIXDB-2501. - Resolution: Fixed > Redo failed at negative length value > > > Key: ASTERIXDB-2501 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2501 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Xikui Wang >Priority: Critical > Attachments: access.log, blue-service.log, cc (2).log, red-service.log > > > A customer who may have shutdown his computer without properly stopping the > AsterixDB fails to start his AsterixDB instance. It seems the redo log has a > negative value issue as shown in the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2501) Redo failed at negative length value
[ https://issues.apache.org/jira/browse/ASTERIXDB-2501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761285#comment-16761285 ] Wail Alkowaileet commented on ASTERIXDB-2501: - I think the bug goes back to 2016: ASTERIXDB-1726 ASTERIXDB-2491 The issue is the int-16 overflow. > Redo failed at negative length value > > > Key: ASTERIXDB-2501 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2501 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Xikui Wang >Priority: Critical > Attachments: access.log, blue-service.log, cc (2).log, red-service.log > > > A customer who may have shutdown his computer without properly stopping the > AsterixDB fails to start his AsterixDB instance. It seems the redo log has a > negative value issue as shown in the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2493) In-memory LSM filter is not thread safe
[
https://issues.apache.org/jira/browse/ASTERIXDB-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-2493:
Description:
To reproduce the issue:
1- Setup a cluster with a single NC and a single partition.
2- Set a breakpoint at
[LSMComponentFilter.java#L71|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMComponentFilter.java#L71]
3- DDL:
{code:sql}
DROP DATAVERSE ThreadSafe IF EXISTS;
CREATE DATAVERSE ThreadSafe;
USE ThreadSafe;
CREATE TYPE FilterTestType AS {
uid: uuid,
created: int
};
CREATE DATASET FilterTest(FilterTestType)
PRIMARY KEY uid AUTOGENERATED WITH FILTER ON created;
{code}
4- Initiate two insert queries:
{code:sql}
USE ThreadSafe;
INSERT INTO FilterTest (
{"created": 1}
)
INSERT INTO FilterTest (
{"created": 0}
)
{code}
5- Let the insert with "created = 0" to update the minTuple (L79)
6- Now, let the insert with "created = 1" to update minTuple
7- Do the same for the max.
After (7) both min and max should equal to 1
8- Flush the component:
[http://localhost:19002/connector?dataverseName=ThreadSafe=FilterTest]
9- Execute search query:
{code:sql}
USE ThreadSafe;
SELECT *
FROM FilterTest
WHERE created = 0;
{code}
The query returns an empty result
was:
To reproduce the issue:
1- Setup a cluster with a single NC and a single partition.
2- Set a breakpoint at
[LSMComponentFilter.java#L71]|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMComponentFilter.java#L71]
3- DDL:
{code:sql}
DROP DATAVERSE ThreadSafe IF EXISTS;
CREATE DATAVERSE ThreadSafe;
USE ThreadSafe;
CREATE TYPE FilterTestType AS {
uid: uuid,
created: int
};
CREATE DATASET FilterTest(FilterTestType)
PRIMARY KEY uid AUTOGENERATED WITH FILTER ON created;
{code}
4- Initiate two insert queries:
{code:sql}
USE ThreadSafe;
INSERT INTO FilterTest (
{"created": 1}
)
INSERT INTO FilterTest (
{"created": 0}
)
{code}
5- Let the insert with "created = 0" to update the minTuple (L79)
6- Now, let the insert with "created = 1" to update minTuple
7- Do the same for the max.
After (7) both min and max should equal to 1
8- Flush the component:
http://localhost:19002/connector?dataverseName=ThreadSafe=FilterTest
9- Execute search query:
{code:sql}
USE ThreadSafe;
SELECT *
FROM FilterTest
WHERE created = 0;
{code}
The query returns an empty result
> In-memory LSM filter is not thread safe
> ---
>
> Key: ASTERIXDB-2493
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2493
> Project: Apache AsterixDB
> Issue Type: Bug
>Reporter: Wail Alkowaileet
>Priority: Major
>
> To reproduce the issue:
> 1- Setup a cluster with a single NC and a single partition.
> 2- Set a breakpoint at
> [LSMComponentFilter.java#L71|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMComponentFilter.java#L71]
> 3- DDL:
> {code:sql}
> DROP DATAVERSE ThreadSafe IF EXISTS;
> CREATE DATAVERSE ThreadSafe;
> USE ThreadSafe;
> CREATE TYPE FilterTestType AS {
> uid: uuid,
> created: int
> };
> CREATE DATASET FilterTest(FilterTestType)
> PRIMARY KEY uid AUTOGENERATED WITH FILTER ON created;
> {code}
> 4- Initiate two insert queries:
> {code:sql}
> USE ThreadSafe;
> INSERT INTO FilterTest (
> {"created": 1}
> )
> INSERT INTO FilterTest (
> {"created": 0}
> )
> {code}
> 5- Let the insert with "created = 0" to update the minTuple (L79)
> 6- Now, let the insert with "created = 1" to update minTuple
> 7- Do the same for the max.
> After (7) both min and max should equal to 1
> 8- Flush the component:
>
> [http://localhost:19002/connector?dataverseName=ThreadSafe=FilterTest]
> 9- Execute search query:
> {code:sql}
> USE ThreadSafe;
> SELECT *
> FROM FilterTest
> WHERE created = 0;
> {code}
> The query returns an empty result
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2493) In-memory LSM filter is not thread safe
Wail Alkowaileet created ASTERIXDB-2493:
---
Summary: In-memory LSM filter is not thread safe
Key: ASTERIXDB-2493
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2493
Project: Apache AsterixDB
Issue Type: Bug
Reporter: Wail Alkowaileet
To reproduce the issue:
1- Setup a cluster with a single NC and a single partition.
2- Set a breakpoint at
[LSMComponentFilter.java#L71]|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/LSMComponentFilter.java#L71]
3- DDL:
{code:sql}
DROP DATAVERSE ThreadSafe IF EXISTS;
CREATE DATAVERSE ThreadSafe;
USE ThreadSafe;
CREATE TYPE FilterTestType AS {
uid: uuid,
created: int
};
CREATE DATASET FilterTest(FilterTestType)
PRIMARY KEY uid AUTOGENERATED WITH FILTER ON created;
{code}
4- Initiate two insert queries:
{code:sql}
USE ThreadSafe;
INSERT INTO FilterTest (
{"created": 1}
)
INSERT INTO FilterTest (
{"created": 0}
)
{code}
5- Let the insert with "created = 0" to update the minTuple (L79)
6- Now, let the insert with "created = 1" to update minTuple
7- Do the same for the max.
After (7) both min and max should equal to 1
8- Flush the component:
http://localhost:19002/connector?dataverseName=ThreadSafe=FilterTest
9- Execute search query:
{code:sql}
USE ThreadSafe;
SELECT *
FROM FilterTest
WHERE created = 0;
{code}
The query returns an empty result
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ASTERIXDB-2492) Skip filter for undo
[ https://issues.apache.org/jira/browse/ASTERIXDB-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet reassigned ASTERIXDB-2492: --- Assignee: Wail Alkowaileet > Skip filter for undo > > > Key: ASTERIXDB-2492 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2492 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Affects Versions: 0.9.5 >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > Attachments: nc-asterix_nc1-12-07-18.log.gz > > > Currently, the RecoveryManager is not skipping the filter for undo. This > means currently the filter tuple is storing arbitrary value when undo > performed (for instance inserting a tuple with duplicate key). > This bug appeared after the change at > [ASTERIXDB-2491|https://issues.apache.org/jira/browse/ASTERIXDB-2491] > Attached log. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2492) Skip filter for undo
Wail Alkowaileet created ASTERIXDB-2492: --- Summary: Skip filter for undo Key: ASTERIXDB-2492 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2492 Project: Apache AsterixDB Issue Type: Bug Components: HYR - Hyracks, TX - Transactions Affects Versions: 0.9.5 Reporter: Wail Alkowaileet Attachments: nc-asterix_nc1-12-07-18.log.gz Currently, the RecoveryManager is not skipping the filter for undo. This means currently the filter tuple is storing arbitrary value when undo performed (for instance inserting a tuple with duplicate key). This bug appeared after the change at [ASTERIXDB-2491|https://issues.apache.org/jira/browse/ASTERIXDB-2491] Attached log. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ASTERIXDB-2491) Recovery fail for large tuples (short integer overflow)
[ https://issues.apache.org/jira/browse/ASTERIXDB-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet reassigned ASTERIXDB-2491: --- Assignee: Wail Alkowaileet > Recovery fail for large tuples (short integer overflow) > --- > > Key: ASTERIXDB-2491 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2491 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Critical > Attachments: cc.log, nc-1.log > > > I've been running some tests with pretty large objects (~100KB). But it seems > the field end offsets in the log buffer are of type short [1, 2]. For large > fields (> 32KB), the field end offset is going to be a negative value due to > overflow. When recovery kicks in, I get an exception that "The length must be > a non-negative value". > [1] > [SimpleTupleWriter|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java#L106] > [2] > [SimpleTupleReference|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java#L86] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2491) Recovery fail for large tuples (short integer overflow)
[ https://issues.apache.org/jira/browse/ASTERIXDB-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-2491: Attachment: nc-1.log cc.log > Recovery fail for large tuples (short integer overflow) > --- > > Key: ASTERIXDB-2491 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2491 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Reporter: Wail Alkowaileet >Priority: Critical > Attachments: cc.log, nc-1.log > > > I've been running some tests with pretty large objects (~100KB). But it seems > the field end offsets in the log buffer are of type short [1, 2]. For large > fields (> 32KB), the field end offset is going to be a negative value due to > overflow. When recovery kicks in, I get an exception that "The length must be > a non-negative value". > [1] > [SimpleTupleWriter|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java#L106] > [2] > [SimpleTupleReference|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java#L86] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2491) Recovery fail for large tuples (short integer overflow)
[ https://issues.apache.org/jira/browse/ASTERIXDB-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713136#comment-16713136 ] Wail Alkowaileet commented on ASTERIXDB-2491: - Attached. I changed it from 2-bytes integer to 4-bytes integer and the problem disappeared. I'm not sure if the change may have a negative implication on current customers. > Recovery fail for large tuples (short integer overflow) > --- > > Key: ASTERIXDB-2491 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2491 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Reporter: Wail Alkowaileet >Priority: Critical > Attachments: cc.log, nc-1.log > > > I've been running some tests with pretty large objects (~100KB). But it seems > the field end offsets in the log buffer are of type short [1, 2]. For large > fields (> 32KB), the field end offset is going to be a negative value due to > overflow. When recovery kicks in, I get an exception that "The length must be > a non-negative value". > [1] > [SimpleTupleWriter|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java#L106] > [2] > [SimpleTupleReference|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java#L86] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2491) Recovery fail for large tuples (short integer overflow)
[ https://issues.apache.org/jira/browse/ASTERIXDB-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-2491: Summary: Recovery fail for large tuples (short integer overflow) (was: Recovery fail for large tuples (short overflow)) > Recovery fail for large tuples (short integer overflow) > --- > > Key: ASTERIXDB-2491 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2491 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, TX - Transactions >Reporter: Wail Alkowaileet >Priority: Critical > > I've been running some tests with pretty large objects (~100KB). But it seems > the field end offsets in the log buffer are of type short [1, 2]. For large > fields (> 32KB), the field end offset is going to be a negative value due to > overflow. When recovery kicks in, I get an exception that "The length must be > a non-negative value". > [1] > [SimpleTupleWriter|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java#L106] > [2] > [SimpleTupleReference|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java#L86] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2491) Recovery fail for large tuples (short overflow)
Wail Alkowaileet created ASTERIXDB-2491: --- Summary: Recovery fail for large tuples (short overflow) Key: ASTERIXDB-2491 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2491 Project: Apache AsterixDB Issue Type: Bug Components: HYR - Hyracks, TX - Transactions Reporter: Wail Alkowaileet I've been running some tests with pretty large objects (~100KB). But it seems the field end offsets in the log buffer are of type short [1, 2]. For large fields (> 32KB), the field end offset is going to be a negative value due to overflow. When recovery kicks in, I get an exception that "The length must be a non-negative value". [1] [SimpleTupleWriter|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java#L106] [2] [SimpleTupleReference|https://github.com/apache/asterixdb/blob/6b31f73565a3b16e0dd1fce9ea010e640c53ca79/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java#L86] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2480) JSON Parser failed at NPE
[
https://issues.apache.org/jira/browse/ASTERIXDB-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684496#comment-16684496
]
Wail Alkowaileet commented on ASTERIXDB-2480:
-
The issue is that the GenericRecord#size() always returns -1 (not the actual
size). I can add a check in the parser to take the whole char[] if
record.size() returns -1.
But it does not sound right to me. Maybe using CharArrayRecord is more suitable
in HttpServerRecordReader?
Thoughts?
> JSON Parser failed at NPE
> -
>
> Key: ASTERIXDB-2480
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2480
> Project: Apache AsterixDB
> Issue Type: Bug
>Affects Versions: 0.9.4
>Reporter: Xikui Wang
>Assignee: Wail Alkowaileet
>Priority: Major
>
> {code}
> drop dataverse lapd if exists;
> create dataverse lapd;
> use lapd;
> create type WorrisomeNotificationType as {
> dataverseName: string,
> channelName: string
> };
> create feed WorrisomeTweetsFeed with {
> "adapter-name" : "http_adapter",
> "addresses" : "127.0.0.1:10011",
> "address-type" : "IP",
> "type-name" : "WorrisomeNotificationType",
> "format" : "json"
> };
> create dataset WorrisomeNotifications(WorrisomeNotificationType) primary key
> dataverseName;
> connect feed WorrisomeTweetsFeed to dataset WorrisomeNotifications;
> start feed WorrisomeTweetsFeed;
> {code}
> feed following json data, a NPE will be thrown.
> { "dataverseName":"dhs", "channelName":"WorrisomeTweetsNearMonumentsChannel",
> "channelExecutionTime":"2018-11-12T18:43:48.508Z"}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (ASTERIXDB-2422) Introduce compressed storage
[ https://issues.apache.org/jira/browse/ASTERIXDB-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet resolved ASTERIXDB-2422. - Resolution: Fixed > Introduce compressed storage > > > Key: ASTERIXDB-2422 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2422 > Project: Apache AsterixDB > Issue Type: Improvement > Components: *DB - AsterixDB, STO - Storage >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > > Details in: > https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ASTERIXDB-2464) createFile() followed by deleteFile() will not unregister the file
[ https://issues.apache.org/jira/browse/ASTERIXDB-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet resolved ASTERIXDB-2464. - Resolution: Fixed > createFile() followed by deleteFile() will not unregister the file > -- > > Key: ASTERIXDB-2464 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2464 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, STO - Storage >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > > In > [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], > we create the file and register it. However, it does not belong to a > FileHandle. > In > [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], > the BufferCache will not unregister the file as it's not in the fileInfoMap. > This probably the reason for the sporadic _the file is already mapped_ issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ASTERIXDB-2464) createFile() followed by deleteFile() will not unregister the file
[ https://issues.apache.org/jira/browse/ASTERIXDB-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet reassigned ASTERIXDB-2464: --- Assignee: Wail Alkowaileet > createFile() followed by deleteFile() will not unregister the file > -- > > Key: ASTERIXDB-2464 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2464 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, STO - Storage >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Major > > In > [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], > we create the file and register it. However, it does not belong to a > FileHandle. > In > [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], > the BufferCache will not unregister the file as it's not in the fileInfoMap. > This probably the reason for the sporadic _the file is already mapped_ issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2464) createFile() followed by deleteFile() will not unregister the file
[ https://issues.apache.org/jira/browse/ASTERIXDB-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-2464: Description: In [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], we create the file and register it. However, it does not belong to a FileHandle. In [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], the BufferCache will not unregister the file as it's not in the fileInfoMap. This probably the reason for the sporadic _the file is already mapped_ issue? was: In [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], we create the file and register it. However, it does not belong to a FileHandle. In [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], the BufferCache will not unregister the file as it's not in the fileInfoMap. This probably the reason for the sporadic issue of the file is already mapped issue? > createFile() followed by deleteFile() will not unregister the file > -- > > Key: ASTERIXDB-2464 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2464 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, STO - Storage >Reporter: Wail Alkowaileet >Priority: Major > > In > [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], > we create the file and register it. However, it does not belong to a > FileHandle. > In > [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], > the BufferCache will not unregister the file as it's not in the fileInfoMap. > This probably the reason for the sporadic _the file is already mapped_ issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2464) createFile() followed by deleteFile() will not unregister the file
[ https://issues.apache.org/jira/browse/ASTERIXDB-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-2464: Description: In [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], we create the file and register it. However, it does not belong to a FileHandle. In [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], the BufferCache will not unregister the file as it's not in the fileInfoMap. This probably the reason for the sporadic issue of the file is already mapped issue? was: In [[1]|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], we create the file and register it. However, it does not belong to a FileHandle. In [[2]|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], the BufferCache will not unregister the file as it's not in the fileInfoMap. This probably the reason for the sporadic issue of the file is already mapped issue? > createFile() followed by deleteFile() will not unregister the file > -- > > Key: ASTERIXDB-2464 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2464 > Project: Apache AsterixDB > Issue Type: Bug > Components: HYR - Hyracks, STO - Storage >Reporter: Wail Alkowaileet >Priority: Major > > In > [BufferCache.java#L817|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], > we create the file and register it. However, it does not belong to a > FileHandle. > In > [BufferCache.java#L1026|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], > the BufferCache will not unregister the file as it's not in the fileInfoMap. > This probably the reason for the sporadic issue of the file is already mapped > issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2464) createFile() followed by deleteFile() will not unregister the file
Wail Alkowaileet created ASTERIXDB-2464: --- Summary: createFile() followed by deleteFile() will not unregister the file Key: ASTERIXDB-2464 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2464 Project: Apache AsterixDB Issue Type: Bug Components: HYR - Hyracks, STO - Storage Reporter: Wail Alkowaileet In [[1]|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L817], we create the file and register it. However, it does not belong to a FileHandle. In [[2]|https://github.com/apache/asterixdb/blob/adfb63361a1808aadb1782aee03acc4d9af8eb0c/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java#L1026], the BufferCache will not unregister the file as it's not in the fileInfoMap. This probably the reason for the sporadic issue of the file is already mapped issue? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2422) Introduce compressed storage
Wail Alkowaileet created ASTERIXDB-2422: --- Summary: Introduce compressed storage Key: ASTERIXDB-2422 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2422 Project: Apache AsterixDB Issue Type: Improvement Components: *DB - AsterixDB, STO - Storage Reporter: Wail Alkowaileet Assignee: Wail Alkowaileet Details in: https://cwiki.apache.org/confluence/display/ASTERIXDB/Compression+in+AsterixDB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2414) FileManager throws file is already mapped on create
[
https://issues.apache.org/jira/browse/ASTERIXDB-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551151#comment-16551151
]
Wail Alkowaileet commented on ASTERIXDB-2414:
-
Reproducer:
In LSMHarness:
- LSMHarness line 575: Add System.exit() in merge(ILSMIOOperation operation)
after doIo(operation) and before exitComponents()
- Run ingestion until exit
- Remove System.exit()
- Start the cluster again.
> FileManager throws file is already mapped on create
> ---
>
> Key: ASTERIXDB-2414
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2414
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB, STO - Storage
>Affects Versions: 0.9.4
>Reporter: Wail Alkowaileet
>Priority: Major
>
> The bug appears in recovery after a hard shutdown during a merge operation.
> {noformat}
> org.apache.hyracks.api.exceptions.HyracksDataException: HYR0081: File
> /Users/wail/asterix-branches/compactor/asterixdb/asterixdb/asterix-app/target/io/dir/asterix_nc1/../asterix-server/target/tmp/asterix_nc1/iodevice2/storage/partition_1/CompactorDataverse/Compactor/0/Compactor/2018-07-16-13-04-17-603_2018-07-16-13-04-09-026_b
> is already mapped
> at
> org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:60)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.common.file.FileMapManager.registerFile(FileMapManager.java:76)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.common.buffercache.BufferCache.createFile(BufferCache.java:815)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.common.impls.AbstractTreeIndex.create(AbstractTreeIndex.java:83)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMDiskComponent.activate(AbstractLSMDiskComponent.java:169)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.api.AbstractLSMWithBloomFilterDiskComponent.activate(AbstractLSMWithBloomFilterDiskComponent.java:54)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.createDiskComponent(AbstractLSMIndex.java:536)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.doMerge(LSMBTree.java:336)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:857)
> ~[classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.doIo(LSMHarness.java:536)
> [classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:575)
> [classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.merge(LSMTreeIndexAccessor.java:127)
> [classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
> [classes/:?]
> at
> org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:1)
> [classes/:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2414) FileManager throws file is already mapped on create
Wail Alkowaileet created ASTERIXDB-2414:
---
Summary: FileManager throws file is already mapped on create
Key: ASTERIXDB-2414
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2414
Project: Apache AsterixDB
Issue Type: Bug
Components: *DB - AsterixDB, STO - Storage
Affects Versions: 0.9.4
Reporter: Wail Alkowaileet
The bug appears in recovery after a hard shutdown during a merge operation.
{noformat}
org.apache.hyracks.api.exceptions.HyracksDataException: HYR0081: File
/Users/wail/asterix-branches/compactor/asterixdb/asterixdb/asterix-app/target/io/dir/asterix_nc1/../asterix-server/target/tmp/asterix_nc1/iodevice2/storage/partition_1/CompactorDataverse/Compactor/0/Compactor/2018-07-16-13-04-17-603_2018-07-16-13-04-09-026_b
is already mapped
at
org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:60)
~[classes/:?]
at
org.apache.hyracks.storage.common.file.FileMapManager.registerFile(FileMapManager.java:76)
~[classes/:?]
at
org.apache.hyracks.storage.common.buffercache.BufferCache.createFile(BufferCache.java:815)
~[classes/:?]
at
org.apache.hyracks.storage.am.common.impls.AbstractTreeIndex.create(AbstractTreeIndex.java:83)
~[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMDiskComponent.activate(AbstractLSMDiskComponent.java:169)
~[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.api.AbstractLSMWithBloomFilterDiskComponent.activate(AbstractLSMWithBloomFilterDiskComponent.java:54)
~[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.createDiskComponent(AbstractLSMIndex.java:536)
~[classes/:?]
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.doMerge(LSMBTree.java:336)
~[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:857)
~[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.doIo(LSMHarness.java:536)
[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:575)
[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.merge(LSMTreeIndexAccessor.java:127)
[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
[classes/:?]
at
org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:1)
[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2331) Plan branch repeated
[
https://issues.apache.org/jira/browse/ASTERIXDB-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401189#comment-16401189
]
Wail Alkowaileet commented on ASTERIXDB-2331:
-
Got it :)
I was confused .. sorry about that.
> Plan branch repeated
>
>
> Key: ASTERIXDB-2331
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2331
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: COMP - Compiler
>Reporter: Wail Alkowaileet
>Assignee: Taewoo Kim
>Priority: Major
>
> I didn't investigate. But it looks like unmaintained Split output.
> DDL
> {noformat}
> DROP DATAVERSE SocialNetworkData IF EXISTS;
> CREATE DATAVERSE SocialNetworkData;
> USE SocialNetworkData;
> create type ChirpMessageType as {
> chirpid: int64,
> send_time: datetime
> };
> create type GleambookUserType as {
> id: int64,
> user_since: datetime
> };
> create type GleambookMessageType as {
> message_id: int64,
> author_id: int64,
> send_time: datetime
> };
> create dataset GleambookMessages(GleambookMessageType)
> primary key message_id;
> create dataset GleambookUsers(GleambookUserType)
> primary key id;
> create dataset ChirpMessages(ChirpMessageType)
> primary key chirpid;
> create index usrSinceIx on GleambookUsers(user_since);
> create index sndTimeIx on ChirpMessages(send_time);
> create index authorIdIx on GleambookMessages(author_id);
> {noformat}
> Query:
> {noformat}
> USE SocialNetworkData;
> EXPLAIN
> SELECT g.message_id
> FROM GleambookUsers as u, GleambookMessages as g
> WHERE u.id/*+indexnl*/ = g.author_id
> AND u.user_since = datetime("2013-04-16T09:45:46")
> {noformat}
> Plan:
> {noformat}
> distribute result [$$28]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> union ($$54, $$55, $$28)
> -- UNION_ALL |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$54])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$54] <- [{"message_id": $$52}]
> -- ASSIGN |PARTITIONED|
> project ([$$52])
> -- STREAM_PROJECT |PARTITIONED|
> select (eq($$29, $$53.getField(1)))
> -- STREAM_SELECT |PARTITIONED|
> project ([$$29, $$52, $$53])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> unnest-map [$$52, $$53] <-
> index-search("GleambookMessages", 0, "SocialNetworkData",
> "GleambookMessages", TRUE, FALSE, 1, $$46, 1, $$46, TRUE, TRUE, TRUE)
> -- BTREE_SEARCH |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$29, $$46])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> split ($$47)
> -- SPLIT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$29, $$46, $$47])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> unnest-map [$$45, $$46, $$47] <-
> index-search("authorIdIx", 0, "SocialNetworkData", "GleambookMessages", TRUE,
> TRUE, 1, $$29, 1, $$29, TRUE, TRUE, TRUE)
> -- BTREE_SEARCH |PARTITIONED|
> exchange
> -- BROADCAST_EXCHANGE |PARTITIONED|
> union ($$43, $$38, $$29)
> -- UNION_ALL |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE
> |PARTITIONED|
> project ([$$43])
> -- STREAM_PROJECT |PARTITIONED|
> select (eq($$33, datetime: {
> 2013-04-16T09:45:46.000Z }))
> -- STREAM_SELECT |PARTITIONED|
> project ([$$43, $$33])
> -- STREAM_PROJECT
> |PARTITIONED|
> assign [$$33] <-
> [$$44.getField(1)]
> -- ASSIGN |PARTITIONED|
> exchange
>
[jira] [Commented] (ASTERIXDB-2331) Plan branch repeated
[
https://issues.apache.org/jira/browse/ASTERIXDB-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398005#comment-16398005
]
Wail Alkowaileet commented on ASTERIXDB-2331:
-
The issue disappeared after set noindexonly 'true';
> Plan branch repeated
>
>
> Key: ASTERIXDB-2331
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2331
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: COMP - Compiler
>Reporter: Wail Alkowaileet
>Priority: Major
>
> I didn't investigate. But it looks like unmaintained Split output.
> DDL
> {noformat}
> DROP DATAVERSE SocialNetworkData IF EXISTS;
> CREATE DATAVERSE SocialNetworkData;
> USE SocialNetworkData;
> create type ChirpMessageType as {
> chirpid: int64,
> send_time: datetime
> };
> create type GleambookUserType as {
> id: int64,
> user_since: datetime
> };
> create type GleambookMessageType as {
> message_id: int64,
> author_id: int64,
> send_time: datetime
> };
> create dataset GleambookMessages(GleambookMessageType)
> primary key message_id;
> create dataset GleambookUsers(GleambookUserType)
> primary key id;
> create dataset ChirpMessages(ChirpMessageType)
> primary key chirpid;
> create index usrSinceIx on GleambookUsers(user_since);
> create index sndTimeIx on ChirpMessages(send_time);
> create index authorIdIx on GleambookMessages(author_id);
> {noformat}
> Query:
> {noformat}
> USE SocialNetworkData;
> EXPLAIN
> SELECT g.message_id
> FROM GleambookUsers as u, GleambookMessages as g
> WHERE u.id/*+indexnl*/ = g.author_id
> AND u.user_since = datetime("2013-04-16T09:45:46")
> {noformat}
> Plan:
> {noformat}
> distribute result [$$28]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> union ($$54, $$55, $$28)
> -- UNION_ALL |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$54])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$54] <- [{"message_id": $$52}]
> -- ASSIGN |PARTITIONED|
> project ([$$52])
> -- STREAM_PROJECT |PARTITIONED|
> select (eq($$29, $$53.getField(1)))
> -- STREAM_SELECT |PARTITIONED|
> project ([$$29, $$52, $$53])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> unnest-map [$$52, $$53] <-
> index-search("GleambookMessages", 0, "SocialNetworkData",
> "GleambookMessages", TRUE, FALSE, 1, $$46, 1, $$46, TRUE, TRUE, TRUE)
> -- BTREE_SEARCH |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$29, $$46])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> split ($$47)
> -- SPLIT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$29, $$46, $$47])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> unnest-map [$$45, $$46, $$47] <-
> index-search("authorIdIx", 0, "SocialNetworkData", "GleambookMessages", TRUE,
> TRUE, 1, $$29, 1, $$29, TRUE, TRUE, TRUE)
> -- BTREE_SEARCH |PARTITIONED|
> exchange
> -- BROADCAST_EXCHANGE |PARTITIONED|
> union ($$43, $$38, $$29)
> -- UNION_ALL |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE
> |PARTITIONED|
> project ([$$43])
> -- STREAM_PROJECT |PARTITIONED|
> select (eq($$33, datetime: {
> 2013-04-16T09:45:46.000Z }))
> -- STREAM_SELECT |PARTITIONED|
> project ([$$43, $$33])
> -- STREAM_PROJECT
> |PARTITIONED|
> assign [$$33] <-
> [$$44.getField(1)]
> -- ASSIGN |PARTITIONED|
> exchange
>
[jira] [Updated] (ASTERIXDB-2331) Plan branch repeated
[
https://issues.apache.org/jira/browse/ASTERIXDB-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-2331:
Description:
I didn't investigate. But it looks like unmaintained Split output.
DDL
{noformat}
DROP DATAVERSE SocialNetworkData IF EXISTS;
CREATE DATAVERSE SocialNetworkData;
USE SocialNetworkData;
create type ChirpMessageType as {
chirpid: int64,
send_time: datetime
};
create type GleambookUserType as {
id: int64,
user_since: datetime
};
create type GleambookMessageType as {
message_id: int64,
author_id: int64,
send_time: datetime
};
create dataset GleambookMessages(GleambookMessageType)
primary key message_id;
create dataset GleambookUsers(GleambookUserType)
primary key id;
create dataset ChirpMessages(ChirpMessageType)
primary key chirpid;
create index usrSinceIx on GleambookUsers(user_since);
create index sndTimeIx on ChirpMessages(send_time);
create index authorIdIx on GleambookMessages(author_id);
{noformat}
Query:
{noformat}
USE SocialNetworkData;
EXPLAIN
SELECT g.message_id
FROM GleambookUsers as u, GleambookMessages as g
WHERE u.id/*+indexnl*/ = g.author_id
AND u.user_since = datetime("2013-04-16T09:45:46")
{noformat}
Plan:
{noformat}
distribute result [$$28]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
union ($$54, $$55, $$28)
-- UNION_ALL |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$54])
-- STREAM_PROJECT |PARTITIONED|
assign [$$54] <- [{"message_id": $$52}]
-- ASSIGN |PARTITIONED|
project ([$$52])
-- STREAM_PROJECT |PARTITIONED|
select (eq($$29, $$53.getField(1)))
-- STREAM_SELECT |PARTITIONED|
project ([$$29, $$52, $$53])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
unnest-map [$$52, $$53] <-
index-search("GleambookMessages", 0, "SocialNetworkData", "GleambookMessages",
TRUE, FALSE, 1, $$46, 1, $$46, TRUE, TRUE, TRUE)
-- BTREE_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$29, $$46])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
split ($$47)
-- SPLIT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$29, $$46, $$47])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
unnest-map [$$45, $$46, $$47] <-
index-search("authorIdIx", 0, "SocialNetworkData", "GleambookMessages", TRUE,
TRUE, 1, $$29, 1, $$29, TRUE, TRUE, TRUE)
-- BTREE_SEARCH |PARTITIONED|
exchange
-- BROADCAST_EXCHANGE |PARTITIONED|
union ($$43, $$38, $$29)
-- UNION_ALL |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$43])
-- STREAM_PROJECT |PARTITIONED|
select (eq($$33, datetime: {
2013-04-16T09:45:46.000Z }))
-- STREAM_SELECT |PARTITIONED|
project ([$$43, $$33])
-- STREAM_PROJECT |PARTITIONED|
assign [$$33] <-
[$$44.getField(1)]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE
|PARTITIONED|
unnest-map [$$43, $$44]
<- index-search("GleambookUsers", 0, "SocialNetworkData", "GleambookUsers",
FALSE, FALSE, 1, $$38, 1, $$38, TRUE, TRUE, TRUE)
-- BTREE_SEARCH
|PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE
|PARTITIONED|
split ($$39)
[jira] [Created] (ASTERIXDB-2331) Plan branch repeated
Wail Alkowaileet created ASTERIXDB-2331:
---
Summary: Plan branch repeated
Key: ASTERIXDB-2331
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2331
Project: Apache AsterixDB
Issue Type: Bug
Components: COMP - Compiler
Reporter: Wail Alkowaileet
I didn't investigate. But it looks like unmaintained Split output.
DDL
{noformat}
DROP DATAVERSE SocialNetworkData IF EXISTS;
CREATE DATAVERSE SocialNetworkData;
USE SocialNetworkData;
create type ChirpMessageType as {
chirpid: int64,
send_time: datetime
};
create type GleambookUserType as {
id: int64,
user_since: datetime
};
create type GleambookMessageType as {
message_id: int64,
author_id: int64,
send_time: datetime
};
create dataset GleambookMessages(GleambookMessageType)
primary key message_id;
create dataset GleambookUsers(GleambookUserType)
primary key id;
create dataset ChirpMessages(ChirpMessageType)
primary key chirpid;
create index usrSinceIx on GleambookUsers(user_since);
create index sndTimeIx on ChirpMessages(send_time);
create index authorIdIx on GleambookMessages(author_id);
{noformat}
Query:
{noformat}
USE SocialNetworkData;
EXPLAIN
SELECT g.message_id
FROM GleambookUsers as u, GleambookMessages as g
WHERE u.id/*+indexnl*/ = g.author_id
AND u.user_since = datetime("2013-04-16T09:45:46")
{noformat}
Plan:
{noformat}
distribute result [$$28]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
union ($$54, $$55, $$28)
-- UNION_ALL |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$54])
-- STREAM_PROJECT |PARTITIONED|
assign [$$54] <- [{"message_id": $$52}]
-- ASSIGN |PARTITIONED|
project ([$$52])
-- STREAM_PROJECT |PARTITIONED|
select (eq($$29, $$53.getField(1)))
-- STREAM_SELECT |PARTITIONED|
project ([$$29, $$52, $$53])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
unnest-map [$$52, $$53] <-
index-search("GleambookMessages", 0, "SocialNetworkData", "GleambookMessages",
TRUE, FALSE, 1, $$46, 1, $$46, TRUE, TRUE, TRUE)
-- BTREE_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$29, $$46])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
split ($$47)
-- SPLIT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$29, $$46, $$47])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
unnest-map [$$45, $$46, $$47] <-
index-search("authorIdIx", 0, "SocialNetworkData", "GleambookMessages", TRUE,
TRUE, 1, $$29, 1, $$29, TRUE, TRUE, TRUE)
-- BTREE_SEARCH |PARTITIONED|
exchange
-- BROADCAST_EXCHANGE |PARTITIONED|
union ($$43, $$38, $$29)
-- UNION_ALL |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$43])
-- STREAM_PROJECT |PARTITIONED|
select (eq($$33, datetime: {
2013-04-16T09:45:46.000Z }))
-- STREAM_SELECT |PARTITIONED|
project ([$$43, $$33])
-- STREAM_PROJECT |PARTITIONED|
assign [$$33] <-
[$$44.getField(1)]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE
|PARTITIONED|
unnest-map [$$43, $$44]
<- index-search("GleambookUsers", 0, "SocialNetworkData", "GleambookUsers",
FALSE, FALSE, 1, $$38, 1, $$38, TRUE, TRUE, TRUE)
-- BTREE_SEARCH
|PARTITIONED|
exchange
[jira] [Closed] (ASTERIXDB-1334) [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records' offset and length on nested ADM types
[
https://issues.apache.org/jira/browse/ASTERIXDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet closed ASTERIXDB-1334.
---
> [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records'
> offset and length on nested ADM types
> ---
>
> Key: ASTERIXDB-1334
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1334
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: TYPE - Data Model
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>Priority: Critical
>
> Hi AsterixDB team,
> There's a problem when I fully define the ADM type. The computed field length
> and offset seem to off.
> To reproduce:
> Download the data:
> https://www.dropbox.com/s/94llrvl08x85z9n/tweet.json?dl=0
> (Sorry it contains Arabic content)
> DDL:
> {noformat}
> create type coordinatesType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type geoType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type metadataType1 as open {
> 'iso_language_code':string,
> 'result_type':string
> }
> create type bounding_boxType1 as open {
> 'coordinates':[[[double]]],
> 'type':string
> }
> create type placeType1 as open {
> 'bounding_box':bounding_boxType1,
> 'country':string,
> 'country_code':string,
> 'full_name':string,
> 'id':string,
> 'name':string,
> 'place_type':string,
> 'url':string
> }
> create type listType1 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'indices':[int64],
> 'url':string
> }
> create type urlType1 as open {
> 'urls':[listType1]
> }
> create type listType2 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type descriptionType1 as open {
> 'urls':[listType2]
> }
> create type entitiesType1 as open {
> 'url':urlType1?,
> 'description':descriptionType1?
> }
> create type userType1 as open {
> 'contributors_enabled':boolean,
> 'created_at':string,
> 'default_profile':boolean,
> 'default_profile_image':boolean,
> 'description':string,
> 'favourites_count':int64,
> 'followers_count':int64,
> 'friends_count':int64,
> 'geo_enabled':boolean,
> 'has_extended_profile':boolean,
> 'id':int64,
> 'id_str':string,
> 'is_translation_enabled':boolean,
> 'is_translator':boolean,
> 'lang':string,
> 'listed_count':int64,
> 'location':string,
> 'name':string,
> 'profile_background_color':string,
> 'profile_background_image_url':string,
> 'profile_background_image_url_https':string,
> 'profile_background_tile':boolean,
> 'profile_banner_url':string?,
> 'profile_image_url':string,
> 'profile_image_url_https':string,
> 'profile_link_color':string,
> 'profile_sidebar_border_color':string,
> 'profile_sidebar_fill_color':string,
> 'profile_text_color':string,
> 'profile_use_background_image':boolean,
> 'protected':boolean,
> 'screen_name':string,
> 'statuses_count':int64,
> 'time_zone':string?,
> 'utc_offset':int64?,
> 'verified':boolean,
> 'entities':entitiesType1?,
> 'url':string?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type listType4 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType5 as open {
> 'id':int64?,
> 'id_str':string?,
> 'indices':[int64]?,
> 'name':string?,
> 'screen_name':string?
> }
> create type largeType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type mediumType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type smallType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type thumbType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type sizesType1 as open {
> 'large':largeType1,
> 'medium':mediumType1,
> 'small':smallType1,
> 'thumb':thumbType1
> }
> create type listType6 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'id':int64,
> 'id_str':string,
> 'indices':[int64],
> 'media_url':string,
> 'media_url_https':string,
> 'sizes':sizesType1,
> 'type':string,
> 'url':string
> }
> create type entitiesType2 as open {
> 'urls':[listType3],
> 'hashtags':[listType4]?,
> 'user_mentions':[listType5]?,
> 'media':[listType6]?
> }
> create type raw_sourceType1 as open {
> 'coordinates':coordinatesType1,
> 'created_at':string,
> 'favorite_count':int64,
>
[jira] [Commented] (ASTERIXDB-1334) [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records' offset and length on nested ADM types
[
https://issues.apache.org/jira/browse/ASTERIXDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384845#comment-16384845
]
Wail Alkowaileet commented on ASTERIXDB-1334:
-
Yes we can.
> [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records'
> offset and length on nested ADM types
> ---
>
> Key: ASTERIXDB-1334
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1334
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: TYPE - Data Model
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>Priority: Critical
>
> Hi AsterixDB team,
> There's a problem when I fully define the ADM type. The computed field length
> and offset seem to off.
> To reproduce:
> Download the data:
> https://www.dropbox.com/s/94llrvl08x85z9n/tweet.json?dl=0
> (Sorry it contains Arabic content)
> DDL:
> {noformat}
> create type coordinatesType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type geoType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type metadataType1 as open {
> 'iso_language_code':string,
> 'result_type':string
> }
> create type bounding_boxType1 as open {
> 'coordinates':[[[double]]],
> 'type':string
> }
> create type placeType1 as open {
> 'bounding_box':bounding_boxType1,
> 'country':string,
> 'country_code':string,
> 'full_name':string,
> 'id':string,
> 'name':string,
> 'place_type':string,
> 'url':string
> }
> create type listType1 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'indices':[int64],
> 'url':string
> }
> create type urlType1 as open {
> 'urls':[listType1]
> }
> create type listType2 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type descriptionType1 as open {
> 'urls':[listType2]
> }
> create type entitiesType1 as open {
> 'url':urlType1?,
> 'description':descriptionType1?
> }
> create type userType1 as open {
> 'contributors_enabled':boolean,
> 'created_at':string,
> 'default_profile':boolean,
> 'default_profile_image':boolean,
> 'description':string,
> 'favourites_count':int64,
> 'followers_count':int64,
> 'friends_count':int64,
> 'geo_enabled':boolean,
> 'has_extended_profile':boolean,
> 'id':int64,
> 'id_str':string,
> 'is_translation_enabled':boolean,
> 'is_translator':boolean,
> 'lang':string,
> 'listed_count':int64,
> 'location':string,
> 'name':string,
> 'profile_background_color':string,
> 'profile_background_image_url':string,
> 'profile_background_image_url_https':string,
> 'profile_background_tile':boolean,
> 'profile_banner_url':string?,
> 'profile_image_url':string,
> 'profile_image_url_https':string,
> 'profile_link_color':string,
> 'profile_sidebar_border_color':string,
> 'profile_sidebar_fill_color':string,
> 'profile_text_color':string,
> 'profile_use_background_image':boolean,
> 'protected':boolean,
> 'screen_name':string,
> 'statuses_count':int64,
> 'time_zone':string?,
> 'utc_offset':int64?,
> 'verified':boolean,
> 'entities':entitiesType1?,
> 'url':string?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type listType4 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType5 as open {
> 'id':int64?,
> 'id_str':string?,
> 'indices':[int64]?,
> 'name':string?,
> 'screen_name':string?
> }
> create type largeType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type mediumType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type smallType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type thumbType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type sizesType1 as open {
> 'large':largeType1,
> 'medium':mediumType1,
> 'small':smallType1,
> 'thumb':thumbType1
> }
> create type listType6 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'id':int64,
> 'id_str':string,
> 'indices':[int64],
> 'media_url':string,
> 'media_url_https':string,
> 'sizes':sizesType1,
> 'type':string,
> 'url':string
> }
> create type entitiesType2 as open {
> 'urls':[listType3],
> 'hashtags':[listType4]?,
> 'user_mentions':[listType5]?,
> 'media':[listType6]?
> }
> create type raw_sourceType1 as open {
> 'coordinates':coordinatesType1,
>
[jira] [Created] (ASTERIXDB-2309) ComponentId failed during redo
Wail Alkowaileet created ASTERIXDB-2309:
---
Summary: ComponentId failed during redo
Key: ASTERIXDB-2309
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2309
Project: Apache AsterixDB
Issue Type: Bug
Reporter: Wail Alkowaileet
Assignee: Chen Luo
Hi,
[~luochen01] I cannot share the data here, but I have a reproducer and I will
provide it to you tomorrow.
{noformat}
java.lang.IllegalStateException: Illegal state of component Id. Max disk
component Id [1519954964710,1519954964710] should be less than redo flush
component Id [1519954964082,1519954964082]
at
org.apache.asterix.app.nc.RecoveryManager.redoFlush(RecoveryManager.java:802)
~[classes/:?]
at
org.apache.asterix.app.nc.RecoveryManager.startRecoveryRedoPhase(RecoveryManager.java:399)
~[classes/:?]
at
org.apache.asterix.app.nc.RecoveryManager.replayPartitionsLogs(RecoveryManager.java:178)
~[classes/:?]
at
org.apache.asterix.app.nc.RecoveryManager.startLocalRecovery(RecoveryManager.java:170)
~[classes/:?]
at
org.apache.asterix.app.nc.task.LocalRecoveryTask.perform(LocalRecoveryTask.java:45)
~[classes/:?]
at
org.apache.asterix.app.replication.message.RegistrationTasksResponseMessage.handle(RegistrationTasksResponseMessage.java:62)
[classes/:?]
at
org.apache.asterix.messaging.NCMessageBroker.lambda$0(NCMessageBroker.java:100)
[classes/:?]
at
org.apache.asterix.messaging.NCMessageBroker$$Lambda$112/1967943797.run(Unknown
Source) [classes/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (ASTERIXDB-2257) Ansible 2.4 throws syntax error
[
https://issues.apache.org/jira/browse/ASTERIXDB-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334844#comment-16334844
]
Wail Alkowaileet edited comment on ASTERIXDB-2257 at 1/22/18 8:30 PM:
--
Yes ... It worked after downgrading to 2.2.1.0. Not sure if it works for 2.3
was (Author: wyk):
Yes ... It worked after downgrading to 2.2.1.0
> Ansible 2.4 throws syntax error
> ---
>
> Key: ASTERIXDB-2257
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2257
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: CLUS - Cluster management
>Reporter: Wail Alkowaileet
>Priority: Minor
>
>
> {noformat}
> ERROR! unexpected parameter type in action:
> The error appears to have been in
> '/asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - name: Start NC Service
> ^ here
> The error appears to have been in
> '/asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - name: Start NC Service
> ^ here
> exception type:
> exception: unexpected parameter type in action:
> The error appears to have been in
> 'asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - name: Start NC Service
> ^ here{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2257) Ansible 2.4 throws syntax error
[
https://issues.apache.org/jira/browse/ASTERIXDB-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334844#comment-16334844
]
Wail Alkowaileet commented on ASTERIXDB-2257:
-
Yes ... It worked after downgrading to 2.2.1.0
> Ansible 2.4 throws syntax error
> ---
>
> Key: ASTERIXDB-2257
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2257
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: CLUS - Cluster management
>Reporter: Wail Alkowaileet
>Priority: Minor
>
>
> {noformat}
> ERROR! unexpected parameter type in action:
> The error appears to have been in
> '/asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - name: Start NC Service
> ^ here
> The error appears to have been in
> '/asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - name: Start NC Service
> ^ here
> exception type:
> exception: unexpected parameter type in action:
> The error appears to have been in
> 'asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - name: Start NC Service
> ^ here{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2253) Disjunctive predicts on the same fields introduces join
[
https://issues.apache.org/jira/browse/ASTERIXDB-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-2253:
Issue Type: Improvement (was: Bug)
> Disjunctive predicts on the same fields introduces join
> ---
>
> Key: ASTERIXDB-2253
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2253
> Project: Apache AsterixDB
> Issue Type: Improvement
> Components: COMP - Compiler
>Reporter: Wail Alkowaileet
>Priority: Major
>
> I'm not sure if I'm missing something ... It looks more expensive than
> StreamSelect
> Query:
> {noformat}
> SELECT value t.text
> FROM Tweets as t
> WHERE t.retweet_count = 10 or t.retweet_count = 20{noformat}
> Plan:
> {noformat}
> distribute result [$$16]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$16])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> join (eq($$19, $$17))
> -- HYBRID_HASH_JOIN [$$17][$$19] |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$16, $$17])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$16, $$17] <- [$$t.getField("text"),
> $$t.getField("retweet_count")]
> -- ASSIGN |PARTITIONED|
> project ([$$t])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$18, $$t] <- TwitterDataverse.Tweets
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
> exchange
> -- BROADCAST_EXCHANGE |PARTITIONED|
> unnest $$19 <- scan-collection(array: [ 20, 10 ])
> -- UNNEST |UNPARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |UNPARTITIONED|
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2257) Ansible 2.4 throws syntax error
Wail Alkowaileet created ASTERIXDB-2257:
---
Summary: Ansible 2.4 throws syntax error
Key: ASTERIXDB-2257
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2257
Project: Apache AsterixDB
Issue Type: Bug
Components: CLUS - Cluster management
Reporter: Wail Alkowaileet
{noformat}
ERROR! unexpected parameter type in action:
The error appears to have been in
'/asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Start NC Service
^ here
The error appears to have been in
'/asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Start NC Service
^ here
exception type:
exception: unexpected parameter type in action:
The error appears to have been in
'asterix/opt/ansible/yaml/start_ncservice.yml': line 20, column 3, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Start NC Service
^ here{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ASTERIXDB-2233) Common conjunctions in disjunctions
[
https://issues.apache.org/jira/browse/ASTERIXDB-2233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet reassigned ASTERIXDB-2233:
---
Assignee: Wail Alkowaileet
> Common conjunctions in disjunctions
> ---
>
> Key: ASTERIXDB-2233
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2233
> Project: Apache AsterixDB
> Issue Type: Improvement
> Components: COMP - Compiler
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>Priority: Major
>
> (Inspired by Apache Impala)
> Boolean expression in the form:
> * (p and q and r) or r -> r
> * (p and q) or (p and q) -> p and q
> This transformation unlocks other optimizations to kick in and it is
> compatible with MISSING/NULL truth table.
> Before:
> Query:
> {noformat}
> SELECT t.x
> FROM Tweets as t, TweetsExt as te
> WHERE (t.x = te.x and te.y > 10) or (t.x = te.x and te.z > 10)
> {noformat}
> Plan:
> {noformat}
> distribute result [$$34]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$34])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$34] <- [{"x": $$38}]
> -- ASSIGN |PARTITIONED|
> project ([$$38])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> join (or(and(eq($$38, $$39), gt($$40, 10)), and(eq($$38, $$39),
> gt($$42, 10
> -- NESTED_LOOP |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$38])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$38] <- [$$t.getField("x")]
> -- ASSIGN |PARTITIONED|
> project ([$$t])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$37, $$t] <- TwitterDataverse.Tweets
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
> exchange
> -- BROADCAST_EXCHANGE |PARTITIONED|
> project ([$$39, $$40, $$42])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$42, $$40, $$39] <- [$$te.getField("z"),
> $$te.getField("y"), $$te.getField("x")]
> -- ASSIGN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$te] <- TwitterDataverse.TweetsExt
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
> {noformat}
> After:
> {noformat}
> SELECT t.x
> FROM Tweets as t, TweetsExt as te
> WHERE t.x = te.x and (te.y > 10 or te.z > 10)
> {noformat}
> Plan:
> {noformat}
> distribute result [$$30]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$30])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$30] <- [{"x": $$31}]
> -- ASSIGN |PARTITIONED|
> project ([$$31])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> join (eq($$31, $$33))
> -- HYBRID_HASH_JOIN [$$31][$$33] |PARTITIONED|
> exchange
> -- HASH_PARTITION_EXCHANGE [$$31] |PARTITIONED|
> project ([$$31])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$31] <- [$$t.getField("x")]
> -- ASSIGN |PARTITIONED|
> project ([$$t])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$32, $$t] <- TwitterDataverse.Tweets
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
> exchange
> -- HASH_PARTITION_EXCHANGE [$$33] |PARTITIONED|
> project ([$$33])
> -- STREAM_PROJECT |PARTITIONED|
> select (or(gt($$te.getField("y"), 10),
> gt($$te.getField("z"), 10)))
> --
[jira] [Commented] (ASTERIXDB-2253) Disjunctive predicts on the same fields introduces join
[
https://issues.apache.org/jira/browse/ASTERIXDB-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332819#comment-16332819
]
Wail Alkowaileet commented on ASTERIXDB-2253:
-
I agree...
Also I found it's needed to rewrite select with disjuncts into index access
method.
> Disjunctive predicts on the same fields introduces join
> ---
>
> Key: ASTERIXDB-2253
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2253
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: COMP - Compiler
>Reporter: Wail Alkowaileet
>Priority: Major
>
> I'm not sure if I'm missing something ... It looks more expensive than
> StreamSelect
> Query:
> {noformat}
> SELECT value t.text
> FROM Tweets as t
> WHERE t.retweet_count = 10 or t.retweet_count = 20{noformat}
> Plan:
> {noformat}
> distribute result [$$16]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$16])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> join (eq($$19, $$17))
> -- HYBRID_HASH_JOIN [$$17][$$19] |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$16, $$17])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$16, $$17] <- [$$t.getField("text"),
> $$t.getField("retweet_count")]
> -- ASSIGN |PARTITIONED|
> project ([$$t])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$18, $$t] <- TwitterDataverse.Tweets
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
> exchange
> -- BROADCAST_EXCHANGE |PARTITIONED|
> unnest $$19 <- scan-collection(array: [ 20, 10 ])
> -- UNNEST |UNPARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |UNPARTITIONED|
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ASTERIXDB-2253) Disjunctive predicts on the same fields introduces join
[
https://issues.apache.org/jira/browse/ASTERIXDB-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-2253:
Component/s: COMP - Compiler
> Disjunctive predicts on the same fields introduces join
> ---
>
> Key: ASTERIXDB-2253
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2253
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: COMP - Compiler
>Reporter: Wail Alkowaileet
>Priority: Major
>
> I'm not sure if I'm missing something ... It looks more expensive than
> StreamSelect
> Query:
> {noformat}
> SELECT value t.text
> FROM Tweets as t
> WHERE t.retweet_count = 10 or t.retweet_count = 20{noformat}
> Plan:
> {noformat}
> distribute result [$$16]
> -- DISTRIBUTE_RESULT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$16])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> join (eq($$19, $$17))
> -- HYBRID_HASH_JOIN [$$17][$$19] |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> project ([$$16, $$17])
> -- STREAM_PROJECT |PARTITIONED|
> assign [$$16, $$17] <- [$$t.getField("text"),
> $$t.getField("retweet_count")]
> -- ASSIGN |PARTITIONED|
> project ([$$t])
> -- STREAM_PROJECT |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> data-scan []<-[$$18, $$t] <- TwitterDataverse.Tweets
> -- DATASOURCE_SCAN |PARTITIONED|
> exchange
> -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |PARTITIONED|
> exchange
> -- BROADCAST_EXCHANGE |PARTITIONED|
> unnest $$19 <- scan-collection(array: [ 20, 10 ])
> -- UNNEST |UNPARTITIONED|
> empty-tuple-source
> -- EMPTY_TUPLE_SOURCE |UNPARTITIONED|
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2254) Recognize common commutative expressions
Wail Alkowaileet created ASTERIXDB-2254:
---
Summary: Recognize common commutative expressions
Key: ASTERIXDB-2254
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2254
Project: Apache AsterixDB
Issue Type: Improvement
Components: COMP - Compiler
Reporter: Wail Alkowaileet
Currently, Algebricks does not recognize common commutative expressions.
Query:
{noformat}
SELECT t.retweet_count + 10, 10 + t.retweet_count
FROM Tweets as t{noformat}
Plan:
{noformat}
distribute result [$$15]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$15])
-- STREAM_PROJECT |PARTITIONED|
assign [$$15] <- [{"$1": numeric-add($$16, 10), "$2": numeric-add(10,
$$16)}]
-- ASSIGN |PARTITIONED|
project ([$$16])
-- STREAM_PROJECT |PARTITIONED|
assign [$$16] <- [$$t.getField("retweet_count")]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$17, $$t] <- TwitterDataverse.Tweets
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ASTERIXDB-2253) Disjunctive predicts on the same fields introduces join
Wail Alkowaileet created ASTERIXDB-2253:
---
Summary: Disjunctive predicts on the same fields introduces join
Key: ASTERIXDB-2253
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2253
Project: Apache AsterixDB
Issue Type: Bug
Reporter: Wail Alkowaileet
I'm not sure if I'm missing something ... It looks more expensive than
StreamSelect
Query:
{noformat}
SELECT value t.text
FROM Tweets as t
WHERE t.retweet_count = 10 or t.retweet_count = 20{noformat}
Plan:
{noformat}
distribute result [$$16]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$16])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (eq($$19, $$17))
-- HYBRID_HASH_JOIN [$$17][$$19] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$16, $$17])
-- STREAM_PROJECT |PARTITIONED|
assign [$$16, $$17] <- [$$t.getField("text"),
$$t.getField("retweet_count")]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$18, $$t] <- TwitterDataverse.Tweets
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- BROADCAST_EXCHANGE |PARTITIONED|
unnest $$19 <- scan-collection(array: [ 20, 10 ])
-- UNNEST |UNPARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |UNPARTITIONED|
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ASTERIXDB-2218) Enforcement of a secondary index does not work.
[
https://issues.apache.org/jira/browse/ASTERIXDB-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315533#comment-16315533
]
Wail Alkowaileet commented on ASTERIXDB-2218:
-
For the closed-type, both the primary index and the secondary index will have
the same *demoted* value (64). I'm not sure if that's a correct behavior or a
type mismatch exception should be thrown.
However for the open-type, the primary index will have double value (64.79) and
the secondary index will get the demoted value.
Maybe we should only allow promote but no demote for Insert/upsert/update?
> Enforcement of a secondary index does not work.
> ---
>
> Key: ASTERIXDB-2218
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2218
> Project: Apache AsterixDB
> Issue Type: Bug
>Reporter: Taewoo Kim
>
> The enforced index does not check the field type when inserting a record. The
> following code work on the current master.
> {code}
> create type tempType if not exists as open {
> id: int64
> };
> create dataset tempDataset(tempType) primary key id;
> create index tempIndex on tempDataset(val:int64?) enforced;
> insert into tempDataset({"id":1,"val":64.79});
> {code}
> For a closed-type field, it works as well.
> {code}
> create type tempClosedType if not exists as closed {
> id: int64,
> val: int64
> };
> create dataset tempClosedDataset(tempClosedType) primary key id;
> create index tempClosedIndex on tempClosedDataset(val);
> insert into tempClosedDataset({"id":1,"val":64.79});
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (ASTERIXDB-2233) Common conjunctions in disjunctions
Wail Alkowaileet created ASTERIXDB-2233:
---
Summary: Common conjunctions in disjunctions
Key: ASTERIXDB-2233
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2233
Project: Apache AsterixDB
Issue Type: Improvement
Components: COMP - Compiler
Reporter: Wail Alkowaileet
(Inspired by Apache Impala)
Boolean expression in the form:
* (p and q and r) or r -> r
* (p and q) or (p and q) -> p and q
This transformation unlocks other optimizations to kick in and it is compatible
with MISSING/NULL truth table.
Before:
Query:
{noformat}
SELECT t.x
FROM Tweets as t, TweetsExt as te
WHERE (t.x = te.x and te.y > 10) or (t.x = te.x and te.z > 10)
{noformat}
Plan:
{noformat}
distribute result [$$34]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$34])
-- STREAM_PROJECT |PARTITIONED|
assign [$$34] <- [{"x": $$38}]
-- ASSIGN |PARTITIONED|
project ([$$38])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (or(and(eq($$38, $$39), gt($$40, 10)), and(eq($$38, $$39),
gt($$42, 10
-- NESTED_LOOP |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$38])
-- STREAM_PROJECT |PARTITIONED|
assign [$$38] <- [$$t.getField("x")]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$37, $$t] <- TwitterDataverse.Tweets
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- BROADCAST_EXCHANGE |PARTITIONED|
project ([$$39, $$40, $$42])
-- STREAM_PROJECT |PARTITIONED|
assign [$$42, $$40, $$39] <- [$$te.getField("z"),
$$te.getField("y"), $$te.getField("x")]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$te] <- TwitterDataverse.TweetsExt
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
After:
{noformat}
SELECT t.x
FROM Tweets as t, TweetsExt as te
WHERE t.x = te.x and (te.y > 10 or te.z > 10)
{noformat}
Plan:
{noformat}
distribute result [$$30]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$30])
-- STREAM_PROJECT |PARTITIONED|
assign [$$30] <- [{"x": $$31}]
-- ASSIGN |PARTITIONED|
project ([$$31])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (eq($$31, $$33))
-- HYBRID_HASH_JOIN [$$31][$$33] |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$31] |PARTITIONED|
project ([$$31])
-- STREAM_PROJECT |PARTITIONED|
assign [$$31] <- [$$t.getField("x")]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$32, $$t] <- TwitterDataverse.Tweets
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$33] |PARTITIONED|
project ([$$33])
-- STREAM_PROJECT |PARTITIONED|
select (or(gt($$te.getField("y"), 10), gt($$te.getField("z"),
10)))
-- STREAM_SELECT |PARTITIONED|
assign [$$33] <- [$$te.getField("x")]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$te] <- TwitterDataverse.TweetsExt
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
[jira] [Created] (ASTERIXDB-2221) EquiJoin condition propagation for constant values
Wail Alkowaileet created ASTERIXDB-2221:
---
Summary: EquiJoin condition propagation for constant values
Key: ASTERIXDB-2221
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2221
Project: Apache AsterixDB
Issue Type: Improvement
Reporter: Wail Alkowaileet
When EquiJoin condition: t1.a=t2.a AND t.a = CONSTANT
The filtration only applied on t1.a but not t2.a
Query:
{noformat}
SELECT t.text as t1, te.text as t2
FROM Tweets as t, TweetsExt as te
WHERE t.a = te.a and t.a = "10"
{noformat}
Plan:
{noformat}
distribute result [$$28]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$28])
-- STREAM_PROJECT |PARTITIONED|
assign [$$28] <- [{"t1": $$32, "t2": $$33}]
-- ASSIGN |PARTITIONED|
project ([$$32, $$33])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (eq($$29, $$31))
-- HYBRID_HASH_JOIN [$$29][$$31] |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$29] |PARTITIONED|
select (eq($$29, "10"))
-- STREAM_SELECT |PARTITIONED|
project ([$$32, $$29])
-- STREAM_PROJECT |PARTITIONED|
assign [$$32, $$29] <- [$$t.getField("text"),
$$t.getField("a")]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$30, $$t] <- TwitterDataverse.Tweets
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$31] |PARTITIONED|
project ([$$33, $$31])
-- STREAM_PROJECT |PARTITIONED|
assign [$$33, $$31] <- [$$te.getField("text"),
$$te.getField("a")]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$te] <- TwitterDataverse.TweetsExt
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (ASTERIXDB-2220) Resolve nested key expression
Wail Alkowaileet created ASTERIXDB-2220:
---
Summary: Resolve nested key expression
Key: ASTERIXDB-2220
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2220
Project: Apache AsterixDB
Issue Type: Bug
Reporter: Wail Alkowaileet
Assignee: Steven Jacobs
key field-access expression is not recognized when the first argument is a
variable
Reproduce
DDL:
{noformat}
DROP DATAVERSE Facebook IF EXISTS;
CREATE DATAVERSE Facebook;
Use Facebook;
CREATE TYPE PersonType AS closed
{ id:string, name:string }
;
CREATE TYPE FriendshipType AS closed
{ person : PersonType, Friends :[PersonType] }
;
/* Creating Datasets */
CREATE DATASET Person(PersonType)
PRIMARY KEY id;
CREATE DATASET Friendship(FriendshipType)
PRIMARY KEY person.id;
{noformat}
Query:
{noformat}
Use Facebook;
select first.person.name as n1, second.person.name as n2
from Friendship first, Friendship second
where first.person.id = second.person.id;
{noformat}
Plan:
{noformat}
"distribute result [$$29]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$29])
-- STREAM_PROJECT |PARTITIONED|
assign [$$29] <- [{"n1": $$36, "n2": $$37}]
-- ASSIGN |PARTITIONED|
project ([$$36, $$37])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (eq($$34, $$35))
-- HYBRID_HASH_JOIN [$$34][$$35] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$36, $$34])
-- STREAM_PROJECT |PARTITIONED|
assign [$$36, $$34] <- [$$37, $$35]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
replicate
-- REPLICATE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$35] |PARTITIONED|
project ([$$37, $$35])
-- STREAM_PROJECT |PARTITIONED|
assign [$$37, $$35] <- [$$31.getField(1),
$$31.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$31])
-- STREAM_PROJECT |PARTITIONED|
assign [$$31] <- [$$second.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$33, $$second] <-
Facebook.Friendship
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
replicate
-- REPLICATE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$35] |PARTITIONED|
project ([$$37, $$35])
-- STREAM_PROJECT |PARTITIONED|
assign [$$37, $$35] <- [$$31.getField(1),
$$31.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$31])
-- STREAM_PROJECT |PARTITIONED|
assign [$$31] <- [$$second.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$33, $$second] <-
Facebook.Friendship
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (ASTERIXDB-2217) deep_equal throws ClassCastException
Wail Alkowaileet created ASTERIXDB-2217:
---
Summary: deep_equal throws ClassCastException
Key: ASTERIXDB-2217
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2217
Project: Apache AsterixDB
Issue Type: Bug
Components: FUN - Functions
Reporter: Wail Alkowaileet
deep_equal doesn't allocate the right pointables for open types.
DDL:
{noformat}
DROP DATAVERSE DeepEqualDataverse IF EXISTS;
CREATE DATAVERSE DeepEqualDataverse;
USE DeepEqualDataverse;
CREATE TYPE EqType as {
a: int
};
CREATE DATASET Eq(EqType)
PRIMARY KEY a;
{noformat}
DML:
{noformat}
USE DeepEqualDataverse;
INSERT INTO Eq(
{"a":1, "b": {"d":3, "c":1}}
)
{noformat}
Query:
{noformat}
USE DeepEqualDataverse;
SELECT *
FROM Eq as e
WHERE deep_equal({"c": 1, "d":3}, e.b)
{noformat}
Output:
{noformat}
Error ClassCastException: org.apache.asterix.om.pointables.AFlatValuePointable
cannot be cast to org.apache.asterix.om.pointables.ARecordVisitablePointable
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ASTERIXDB-2199) Nested primary key and hash repartitioning bug
[
https://issues.apache.org/jira/browse/ASTERIXDB-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306655#comment-16306655
]
Wail Alkowaileet commented on ASTERIXDB-2199:
-
Yes, similar to the issue in this JIRA. However, the fix doesn't recognize the
key expression if it's in the form
{noformat}
$assignVar.getField(0)
{noformat}
where $assingVar = $unnestVar.getField(0)
it only recognizes it if it is inlined nested field access
{noformat}
$unnestVar.getField(0).getField(0)
{noformat}
where $unnestVar is the scan variable from data-scan
> Nested primary key and hash repartitioning bug
> ---
>
> Key: ASTERIXDB-2199
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2199
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
>Reporter: Shiva Jahangiri
>Assignee: Steven Jacobs
>
> If a join is happening on primary keys of two tables, no hash partitioning
> should happen. Having the following DDL(Note that primary key of Friendship2
> is string):
> DROP DATAVERSE Facebook IF EXISTS;
> CREATE DATAVERSE Facebook;
> Use Facebook;
> CREATE TYPE FriendshipType AS closed {
> id:string,
> friends :[string]
> };
> CREATE DATASET Friendship2(FriendshipType)
> PRIMARY KEY id;
> insert into Friendship2([ {"id":"1","friends" : [ "2","3","4"]},
> {"id":"2","friends" : [ "4","5","6"]}
> ]);
> By running the following query:
> Use Facebook;
> select * from Friendship2 first, Friendship2 second where first.id =
> second.id;
> we can see that there is no hash partitioning happening in optimized logical
> plan which is correct as join is happening on the primary key of both
> relations and data is already partitioned on primary key:
> {
> "operator":"distribute-result",
> "expressions":"$$9",
> "operatorId" : "1.1",
> "physical-operator":"DISTRIBUTE_RESULT",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.2",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
>"operator":"project",
>"variables" :["$$9"],
>"operatorId" : "1.3",
>"physical-operator":"STREAM_PROJECT",
>"execution-mode":"PARTITIONED",
>"inputs":[
>{
> "operator":"assign",
> "variables" :["$$9"],
> "expressions":"{ first : $$first, second : $$second}",
> "operatorId" : "1.4",
> "physical-operator":"ASSIGN",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"project",
> "variables" :["$$first","$$second"],
> "operatorId" : "1.5",
> "physical-operator":"STREAM_PROJECT",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.6",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
>"operator":"join",
>"condition":"eq($$10, $$11)",
>"operatorId" : "1.7",
>"physical-operator":"HYBRID_HASH_JOIN
> [$$10][$$11]",
>"execution-mode":"PARTITIONED",
>"inputs":[
>{
> "operator":"exchange",
> "operatorId" : "1.8",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"data-scan",
> "variables" :["$$10","$$first"],
> "data-source":"Facebook.Friendship2",
> "operatorId" : "1.9",
>
> "physical-operator":"DATASOURCE_SCAN",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.10",
>
[jira] [Commented] (ASTERIXDB-2199) Nested primary key and hash repartitioning bug
[
https://issues.apache.org/jira/browse/ASTERIXDB-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305941#comment-16305941
]
Wail Alkowaileet commented on ASTERIXDB-2199:
-
[~dtabass] It's a different issue as Steven pointed out...
[~sjaco002] I found another bug in the current fix in
(https://asterix-gerrit.ics.uci.edu/#/c/2246/). I'm not sure if I should file
it as a different issue.
The issue is when the there's a common expression with the key getField()
expression and the key getField() is no longer inlined.
Reproduce
DDL:
{noformat}
DROP DATAVERSE Facebook IF EXISTS;
CREATE DATAVERSE Facebook;
Use Facebook;
CREATE TYPE PersonType AS closed
{ id:string, name:string }
;
CREATE TYPE FriendshipType AS closed
{ person : PersonType, Friends :[PersonType] }
;
/* Creating Datasets */
CREATE DATASET Person(PersonType)
PRIMARY KEY id;
CREATE DATASET Friendship(FriendshipType)
PRIMARY KEY person.id;
{noformat}
Query:
{noformat}
Use Facebook;
select first.person.name as n1, second.person.name as n2
from Friendship first, Friendship second
where first.person.id = second.person.id;
{noformat}
Plan:
{noformat}
"distribute result [$$29]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$29])
-- STREAM_PROJECT |PARTITIONED|
assign [$$29] <- [{"n1": $$36, "n2": $$37}]
-- ASSIGN |PARTITIONED|
project ([$$36, $$37])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (eq($$34, $$35))
-- HYBRID_HASH_JOIN [$$34][$$35] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$36, $$34])
-- STREAM_PROJECT |PARTITIONED|
assign [$$36, $$34] <- [$$37, $$35]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
replicate
-- REPLICATE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$35] |PARTITIONED|
project ([$$37, $$35])
-- STREAM_PROJECT |PARTITIONED|
assign [$$37, $$35] <- [$$31.getField(1),
$$31.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$31])
-- STREAM_PROJECT |PARTITIONED|
assign [$$31] <- [$$second.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$33, $$second] <-
Facebook.Friendship
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
replicate
-- REPLICATE |PARTITIONED|
exchange
-- HASH_PARTITION_EXCHANGE [$$35] |PARTITIONED|
project ([$$37, $$35])
-- STREAM_PROJECT |PARTITIONED|
assign [$$37, $$35] <- [$$31.getField(1),
$$31.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$31])
-- STREAM_PROJECT |PARTITIONED|
assign [$$31] <- [$$second.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$33, $$second] <-
Facebook.Friendship
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
> Nested primary key and hash repartitioning bug
> ---
>
> Key: ASTERIXDB-2199
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2199
> Project: Apache AsterixDB
> Issue Type: Bug
>
[jira] [Commented] (ASTERIXDB-2199) Nested primary key and hash repartitioning bug
[
https://issues.apache.org/jira/browse/ASTERIXDB-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305872#comment-16305872
]
Wail Alkowaileet commented on ASTERIXDB-2199:
-
Will look into it..
Thanks!
> Nested primary key and hash repartitioning bug
> ---
>
> Key: ASTERIXDB-2199
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2199
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
>Reporter: Shiva Jahangiri
>Assignee: Steven Jacobs
>
> If a join is happening on primary keys of two tables, no hash partitioning
> should happen. Having the following DDL(Note that primary key of Friendship2
> is string):
> DROP DATAVERSE Facebook IF EXISTS;
> CREATE DATAVERSE Facebook;
> Use Facebook;
> CREATE TYPE FriendshipType AS closed {
> id:string,
> friends :[string]
> };
> CREATE DATASET Friendship2(FriendshipType)
> PRIMARY KEY id;
> insert into Friendship2([ {"id":"1","friends" : [ "2","3","4"]},
> {"id":"2","friends" : [ "4","5","6"]}
> ]);
> By running the following query:
> Use Facebook;
> select * from Friendship2 first, Friendship2 second where first.id =
> second.id;
> we can see that there is no hash partitioning happening in optimized logical
> plan which is correct as join is happening on the primary key of both
> relations and data is already partitioned on primary key:
> {
> "operator":"distribute-result",
> "expressions":"$$9",
> "operatorId" : "1.1",
> "physical-operator":"DISTRIBUTE_RESULT",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.2",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
>"operator":"project",
>"variables" :["$$9"],
>"operatorId" : "1.3",
>"physical-operator":"STREAM_PROJECT",
>"execution-mode":"PARTITIONED",
>"inputs":[
>{
> "operator":"assign",
> "variables" :["$$9"],
> "expressions":"{ first : $$first, second : $$second}",
> "operatorId" : "1.4",
> "physical-operator":"ASSIGN",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"project",
> "variables" :["$$first","$$second"],
> "operatorId" : "1.5",
> "physical-operator":"STREAM_PROJECT",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.6",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
>"operator":"join",
>"condition":"eq($$10, $$11)",
>"operatorId" : "1.7",
>"physical-operator":"HYBRID_HASH_JOIN
> [$$10][$$11]",
>"execution-mode":"PARTITIONED",
>"inputs":[
>{
> "operator":"exchange",
> "operatorId" : "1.8",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"data-scan",
> "variables" :["$$10","$$first"],
> "data-source":"Facebook.Friendship2",
> "operatorId" : "1.9",
>
> "physical-operator":"DATASOURCE_SCAN",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.10",
>
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
>
> "operator":"empty-tuple-source",
>
[jira] [Commented] (ASTERIXDB-2199) Nested primary key and hash repartitioning bug
[
https://issues.apache.org/jira/browse/ASTERIXDB-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304360#comment-16304360
]
Wail Alkowaileet commented on ASTERIXDB-2199:
-
I noticed the same issue... I think the issue is in:
{noformat} EquivalenceClassUtils.addEquivalenceClassesForPrimaryIndexAccess()
{noformat}
It assumes:
{noformat} $second.getField(0) {noformat}
is the primary key not
{noformat} $second.getField(0).getField(0) {noformat}
Query using the same DDL:
{noformat}
USE Facebook;
EXPLAIN
SELECT *
FROM Friendship first, Friendship second
WHERE first.person = second.person;
{noformat}
Plan shows no HASH_PARTITION_EXCHANGE?
{noformat}
distribute result [$$23]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$23])
-- STREAM_PROJECT |PARTITIONED|
assign [$$23] <- [{"first": $$first, "second": $$second}]
-- ASSIGN |PARTITIONED|
project ([$$first, $$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
join (eq($$26, $$27))
-- HYBRID_HASH_JOIN [$$26][$$27] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$first, $$26])
-- STREAM_PROJECT |PARTITIONED|
assign [$$first, $$26] <- [$$second, $$27]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
replicate
-- REPLICATE |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
assign [$$27] <- [$$second.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$25, $$second] <-
Facebook.Friendship
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
replicate
-- REPLICATE |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
assign [$$27] <- [$$second.getField(0)]
-- ASSIGN |PARTITIONED|
project ([$$second])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$25, $$second] <- Facebook.Friendship
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
> Nested primary key and hash repartitioning bug
> ---
>
> Key: ASTERIXDB-2199
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2199
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: *DB - AsterixDB
>Reporter: Shiva Jahangiri
>Assignee: Steven Jacobs
>
> If a join is happening on primary keys of two tables, no hash partitioning
> should happen. Having the following DDL(Note that primary key of Friendship2
> is string):
> DROP DATAVERSE Facebook IF EXISTS;
> CREATE DATAVERSE Facebook;
> Use Facebook;
> CREATE TYPE FriendshipType AS closed {
> id:string,
> friends :[string]
> };
> CREATE DATASET Friendship2(FriendshipType)
> PRIMARY KEY id;
> insert into Friendship2([ {"id":"1","friends" : [ "2","3","4"]},
> {"id":"2","friends" : [ "4","5","6"]}
> ]);
> By running the following query:
> Use Facebook;
> select * from Friendship2 first, Friendship2 second where first.id =
> second.id;
> we can see that there is no hash partitioning happening in optimized logical
> plan which is correct as join is happening on the primary key of both
> relations and data is already partitioned on primary key:
> {
> "operator":"distribute-result",
> "expressions":"$$9",
> "operatorId" : "1.1",
> "physical-operator":"DISTRIBUTE_RESULT",
> "execution-mode":"PARTITIONED",
> "inputs":[
> {
> "operator":"exchange",
> "operatorId" : "1.2",
> "physical-operator":"ONE_TO_ONE_EXCHANGE",
>
[jira] [Created] (ASTERIXDB-2191) Constant folding for nested Scalar Functions
Wail Alkowaileet created ASTERIXDB-2191:
---
Summary: Constant folding for nested Scalar Functions
Key: ASTERIXDB-2191
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2191
Project: Apache AsterixDB
Issue Type: Improvement
Components: COMP - Compiler
Reporter: Wail Alkowaileet
Assignee: Wail Alkowaileet
Currently, ConstantFoldingRule ignores functions that have non-constants as
their arguments. That can miss some opportunities as below:
* Query:
{noformat}
SELECT t
FROM Tweets as t
WHERE t.x = t.x;
{noformat}
* Plan
{noformat}
distribute result [$$7]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$7])
-- STREAM_PROJECT |PARTITIONED|
assign [$$7] <- [{"t": $$t}]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
select (eq($$8, $$8))
-- STREAM_SELECT |PARTITIONED|
assign [$$8] <- [$$t.getField("x")]
-- ASSIGN |PARTITIONED|
project ([$$t])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$9, $$t] <- TwitterDataverse.Tweets
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (ASTERIXDB-2189) AbstractListBuilder.reset() creates unnecessary objects
Wail Alkowaileet created ASTERIXDB-2189: --- Summary: AbstractListBuilder.reset() creates unnecessary objects Key: ASTERIXDB-2189 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2189 Project: Apache AsterixDB Issue Type: Improvement Reporter: Wail Alkowaileet Assignee: Wail Alkowaileet >From [~amoudi]: the reset() for the ordered list builder should be fixed. We nullify a byte array and recreate it with each array. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ASTERIXDB-2041) AQL QueryServiceServlet returns wrong URL handle
Wail Alkowaileet created ASTERIXDB-2041:
---
Summary: AQL QueryServiceServlet returns wrong URL handle
Key: ASTERIXDB-2041
URL: https://issues.apache.org/jira/browse/ASTERIXDB-2041
Project: Apache AsterixDB
Issue Type: Bug
Components: API - HTTP API
Reporter: Wail Alkowaileet
Hello,
To reproduce:
{noformat}
curl --data mode=deferred --data statement=1 http://localhost:19002/query/aql
{noformat}
Response
{noformat}
"requestID": "e2d25082-d1d3-409a-b12c-eded1b8026f5",
"signature": "*",
"handle": "http://localhost:19002/query/aql/result/18-0;,
"status": "success",
"metrics": {
"elapsedTime": "9.296581ms",
"executionTime": "9.045096ms",
"resultCount": 0,
"resultSize": 0
}
{noformat}
Instead of:
http://localhost:19002/query/aql/result/18-0
Should be:
http://localhost:19002/query/service/result/18-0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ASTERIXDB-1885) Printer: Malformed result in closed object with MISSING fields
[
https://issues.apache.org/jira/browse/ASTERIXDB-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974198#comment-15974198
]
Wail Alkowaileet commented on ASTERIXDB-1885:
-
(2) is definitely not valid.
For (1), I was confused a little bit on why we store it. But the same can be
said on why we store empty strings. It's up to the user/data modeler.
Thanks!
> Printer: Malformed result in closed object with MISSING fields
> --
>
> Key: ASTERIXDB-1885
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1885
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: AsterixDB
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>
> I got a problem when dealing with closed objects/records that contain MISSING
> fields.
> Examples:
> {noformat}
> 1: { }
> 2: {,"name":"Alice"}
> {noformat}
> I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
> it's an expected behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (ASTERIXDB-1885) Printer: Malformed result in closed object with MISSING fields
[
https://issues.apache.org/jira/browse/ASTERIXDB-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-1885:
Description:
I got a problem when dealing with closed objects/records that contains MISSING
fields.
Examples:
{noformat}
1: { }
2: {,"name":"Alice"}
{noformat}
I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
it's an expected behavior.
was:
I got a problem when dealing with objects/records that contains MISSING fields.
Examples:
{noformat}
1: { }
2: {,"name":"Alice"}
{noformat}
I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
it's an expected behavior.
> Printer: Malformed result in closed object with MISSING fields
> --
>
> Key: ASTERIXDB-1885
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1885
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: AsterixDB
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>
> I got a problem when dealing with closed objects/records that contains
> MISSING fields.
> Examples:
> {noformat}
> 1: { }
> 2: {,"name":"Alice"}
> {noformat}
> I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
> it's an expected behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (ASTERIXDB-1885) Printer: Malformed result in closed object with MISSING fields
[
https://issues.apache.org/jira/browse/ASTERIXDB-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-1885:
Description:
I got a problem when dealing with closed objects/records that contain MISSING
fields.
Examples:
{noformat}
1: { }
2: {,"name":"Alice"}
{noformat}
I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
it's an expected behavior.
was:
I got a problem when dealing with closed objects/records that contains MISSING
fields.
Examples:
{noformat}
1: { }
2: {,"name":"Alice"}
{noformat}
I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
it's an expected behavior.
> Printer: Malformed result in closed object with MISSING fields
> --
>
> Key: ASTERIXDB-1885
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1885
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: AsterixDB
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>
> I got a problem when dealing with closed objects/records that contain MISSING
> fields.
> Examples:
> {noformat}
> 1: { }
> 2: {,"name":"Alice"}
> {noformat}
> I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
> it's an expected behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (ASTERIXDB-1885) Malformed result in closed object with MISSING fields
[
https://issues.apache.org/jira/browse/ASTERIXDB-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-1885:
Component/s: AsterixDB
> Malformed result in closed object with MISSING fields
> -
>
> Key: ASTERIXDB-1885
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1885
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: AsterixDB
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>
> I got a problem when dealing with objects/records that contains MISSING
> fields.
> Examples:
> {noformat}
> 1: { }
> 2: {,"name":"Alice"}
> {noformat}
> I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
> it's an expected behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (ASTERIXDB-1885) Printer: Malformed result in closed object with MISSING fields
[
https://issues.apache.org/jira/browse/ASTERIXDB-1885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-1885:
Summary: Printer: Malformed result in closed object with MISSING fields
(was: Malformed result in closed object with MISSING fields)
> Printer: Malformed result in closed object with MISSING fields
> --
>
> Key: ASTERIXDB-1885
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1885
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: AsterixDB
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>
> I got a problem when dealing with objects/records that contains MISSING
> fields.
> Examples:
> {noformat}
> 1: { }
> 2: {,"name":"Alice"}
> {noformat}
> I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
> it's an expected behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (ASTERIXDB-1885) Malformed result in closed object with MISSING fields
Wail Alkowaileet created ASTERIXDB-1885:
---
Summary: Malformed result in closed object with MISSING fields
Key: ASTERIXDB-1885
URL: https://issues.apache.org/jira/browse/ASTERIXDB-1885
Project: Apache AsterixDB
Issue Type: Bug
Reporter: Wail Alkowaileet
Assignee: Wail Alkowaileet
I got a problem when dealing with objects/records that contains MISSING fields.
Examples:
{noformat}
1: { }
2: {,"name":"Alice"}
{noformat}
I fixed the commas in (2). However, I'm not sure if (1) is still valid or if
it's an expected behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (ASTERIXDB-1843) Builtin functions documentation have wrong name and signature for AQL
[ https://issues.apache.org/jira/browse/ASTERIXDB-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-1843: Summary: Builtin functions documentation have wrong name and signature for AQL (was: Builtin functions documentation have wrong name and signature) > Builtin functions documentation have wrong name and signature for AQL > - > > Key: ASTERIXDB-1843 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1843 > Project: Apache AsterixDB > Issue Type: Bug > Components: Documentation, Functions - AQL >Reporter: Wail Alkowaileet > > It seems that the Builtin functions between SQL++ and AQL has been mixed. > For example, > string-concat([String]) vs concat(String, String,...) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (ASTERIXDB-1843) Builtin functions documentation have wrong name and signature
[ https://issues.apache.org/jira/browse/ASTERIXDB-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-1843: Component/s: Documentation > Builtin functions documentation have wrong name and signature > - > > Key: ASTERIXDB-1843 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1843 > Project: Apache AsterixDB > Issue Type: Bug > Components: Documentation, Functions - AQL >Reporter: Wail Alkowaileet > > It seems that the Builtin functions between SQL++ and AQL has been mixed. > For example, > string-concat([String]) vs concat(String, String,...) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (ASTERIXDB-1844) Add START FEED statement
Wail Alkowaileet created ASTERIXDB-1844: --- Summary: Add START FEED statement Key: ASTERIXDB-1844 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1844 Project: Apache AsterixDB Issue Type: Bug Components: Documentation, Feeds Reporter: Wail Alkowaileet Currently AsterixDB doesn't start the ingestion right after CONNECT. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (ASTERIXDB-1843) Builtin functions documentation have wrong name and signature
Wail Alkowaileet created ASTERIXDB-1843: --- Summary: Builtin functions documentation have wrong name and signature Key: ASTERIXDB-1843 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1843 Project: Apache AsterixDB Issue Type: Bug Components: Functions - AQL Reporter: Wail Alkowaileet It seems that the Builtin functions between SQL++ and AQL has been mixed. For example, string-concat([String]) vs concat(String, String,...) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (ASTERIXDB-1369) Add the Java driver for Asterix
[ https://issues.apache.org/jira/browse/ASTERIXDB-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864692#comment-15864692 ] Wail Alkowaileet edited comment on ASTERIXDB-1369 at 2/13/17 11:42 PM: --- Do we have any expectation for what should AsterixDB Java driver have? was (Author: wyk): Do we have any expectation for what should AsterixDB Java driver has? > Add the Java driver for Asterix > --- > > Key: ASTERIXDB-1369 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1369 > Project: Apache AsterixDB > Issue Type: Improvement > Components: Adapters >Reporter: Jianfeng Jia >Priority: Minor > Labels: gsoc2017 > > Add a java driver for AsterixDB. It will be easier for the developer to talk > to the Asterixdb through API than send http request. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (ASTERIXDB-1369) Add the Java driver for Asterix
[ https://issues.apache.org/jira/browse/ASTERIXDB-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864692#comment-15864692 ] Wail Alkowaileet commented on ASTERIXDB-1369: - Do we have any expectation for what should AsterixDB Java driver has? > Add the Java driver for Asterix > --- > > Key: ASTERIXDB-1369 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1369 > Project: Apache AsterixDB > Issue Type: Improvement > Components: Adapters >Reporter: Jianfeng Jia >Priority: Minor > Labels: gsoc2017 > > Add a java driver for AsterixDB. It will be easier for the developer to talk > to the Asterixdb through API than send http request. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (ASTERIXDB-1705) Requesting unexisted handle stops Query API.
Wail Alkowaileet created ASTERIXDB-1705:
---
Summary: Requesting unexisted handle stops Query API.
Key: ASTERIXDB-1705
URL: https://issues.apache.org/jira/browse/ASTERIXDB-1705
Project: Apache AsterixDB
Issue Type: Bug
Components: HTTP API
Reporter: Wail Alkowaileet
Priority: Minor
To reproduce:
Start the cluster and request:
{noformat}
http://localhost:19002/query/status?handle={%22handle%22:[1,0]}
{noformat}
Running any query after thal will not get you any response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1698) Secondary index doesn't follow the compaction policy
[
https://issues.apache.org/jira/browse/ASTERIXDB-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588022#comment-15588022
]
Wail Alkowaileet commented on ASTERIXDB-1698:
-
I had a discussion with Sattam about this. I don't think it's a bug but
unimplemented logic of that specific case.
In the normal case of an LSM index, it will seek the opportunity to have one
disk component as much as possible. So when you create a secondary index, it
will use the bulk loader for building the index.
> Secondary index doesn't follow the compaction policy
>
>
> Key: ASTERIXDB-1698
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1698
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Environment: master : 4819ea44723b87a68406d248782861cf6e5d3305
>Reporter: Jianfeng Jia
>Assignee: Ian Maxon
>
> Here is the ddl for the dataset:
> {code}
> create dataset ds_tweet(typeTweet) if not exists primary key id using
> compaction policy prefix
> (("max-mergable-component-size"="134217728"),("max-tolerance-component-count"="10"))
> with filter on create_at ;
> create index text_idx if not exists on ds_tweet("text") type keyword;
> {code}
> In this case, I want to create a smaller component around 128M. During the
> data ingestion phase, it works well, and the size of each text_idx component
> is also small (~80M each). I assume it also followed the component size
> constraint?
> After ingestion, I found that I needed to build another index,
> {code}
> create index time_idx if not exists on ds_tweet(create_at) type btree;
> {code}
> When it finished, I found that this time_idx didn't follow the constraint and
> ended up with one giant 1.2G component on each partition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1694) Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes)
[ https://issues.apache.org/jira/browse/ASTERIXDB-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576516#comment-15576516 ] Wail Alkowaileet commented on ASTERIXDB-1694: - After a long meeting, Mingda and I located the problem. Log4J in Twitter4j doesn't initialize properly. I don't know why that's the case. I suspect that there's something wrong in the deployment of AsterixDB. > Fail running Tweet Feed on Cluster of 16 nodes (while succeed on 4 nodes) > - > > Key: ASTERIXDB-1694 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1694 > Project: Apache AsterixDB > Issue Type: Bug > Components: Feeds > Environment: asterix-server-0.8.9-SNAPSHOT-binary-assembly run on > cluster >Reporter: Mingda Li >Assignee: Xikui Wang > Original Estimate: 24h > Remaining Estimate: 24h > > Recently, I am trying to run the data feed query on a cluster of 16 nodes. > All the query run well without error. But each time after I disconnect the > feed, there is none of tweet data stored in dataverse. However, when I try to > run other AQL queries, the cluster can work well. And I have ever used 4 > nodes cluster to load Tweet data successfully. I also checked the log file > and find no error there. This is wired. Does anyone know why? Has anyone ever > used the data feed function on a cluster of 16 nodes or more? > I am using a asterix-server-0.8.9-SNAPSHOT-binary-assembly to configure > cluster compiled by myself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ASTERIXDB-1684) Unify accessing and processing records.
[ https://issues.apache.org/jira/browse/ASTERIXDB-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-1684: Description: Currently, we have more that 5 classes to deal with serialized records: 1- RecordFieldUtils. 2- ARecordSerializerDeserailzer. 3- ARecordPointable. 4- ARecordVisitablePointable. 5- RecordBuilder. Any changes in the binary layout will require to deal with all 5 classes. it would be good if we unify dealing with serialized format in one class. And all other classes use it. was: Currently, we have more that 4 classes to deal with serialized records: 1- RecordFieldUtils. 2- ARecordSerializerDeserailzer. 3- ARecordPointable. 4- ARecordVisitablePointable. 5- RecordBuilder. Any changes in the binary layout will require to deal with all 5 classes. it would be good if we unify dealing with serialized format in one class. And all other classes use it. > Unify accessing and processing records. > --- > > Key: ASTERIXDB-1684 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1684 > Project: Apache AsterixDB > Issue Type: Improvement > Components: Data Model >Reporter: Wail Alkowaileet >Priority: Minor > > Currently, we have more that 5 classes to deal with serialized records: > 1- RecordFieldUtils. > 2- ARecordSerializerDeserailzer. > 3- ARecordPointable. > 4- ARecordVisitablePointable. > 5- RecordBuilder. > Any changes in the binary layout will require to deal with all 5 classes. it > would be good if we unify dealing with serialized format in one class. And > all other classes use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ASTERIXDB-1684) Unify accessing and processing records.
Wail Alkowaileet created ASTERIXDB-1684: --- Summary: Unify accessing and processing records. Key: ASTERIXDB-1684 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1684 Project: Apache AsterixDB Issue Type: Improvement Components: Data Model Reporter: Wail Alkowaileet Priority: Minor Currently, we have more that 4 classes to deal with serialized records: 1- RecordFieldUtils. 2- ARecordSerializerDeserailzer. 3- ARecordPointable. 4- ARecordVisitablePointable. 5- RecordBuilder. Any changes in the binary layout will require to deal with all 5 classes. it would be good if we unify dealing with serialized format in one class. And all other classes use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1670) Java heap exceeded - COUNT(for $x in dataset Tweets return $x)
[
https://issues.apache.org/jira/browse/ASTERIXDB-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554323#comment-15554323
]
Wail Alkowaileet commented on ASTERIXDB-1670:
-
I couldn't reproduce this issue again for some reason.
I'll mark it as not a problem for now.
> Java heap exceeded - COUNT(for $x in dataset Tweets return $x)
> --
>
> Key: ASTERIXDB-1670
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1670
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Optimizer
>Reporter: Wail Alkowaileet
>
> Sounds like there is a new materialization bug.
> [~wyk] Can you please add the query plan for the query that failed with
> Exception.
> {noformat}
> 1- returning the whole tuple
> count( for $x in dataset Tweets
> return $x
> )
> => Throws an exception Java heap exceeded. (The heap-size is less than the
> sum of AsterixDB configured memory ... so it's not a problem).
> {noformat}
> {noformat}
> 2- However, returning one field
> count( for $x in dataset Tweets
> return $x.id
> )
> => Worked just fine.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (ASTERIXDB-1670) Java heap exceeded - COUNT(for $x in dataset Tweets return $x)
[
https://issues.apache.org/jira/browse/ASTERIXDB-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet closed ASTERIXDB-1670.
---
Resolution: Not A Problem
> Java heap exceeded - COUNT(for $x in dataset Tweets return $x)
> --
>
> Key: ASTERIXDB-1670
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1670
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Optimizer
>Reporter: Wail Alkowaileet
>
> Sounds like there is a new materialization bug.
> [~wyk] Can you please add the query plan for the query that failed with
> Exception.
> {noformat}
> 1- returning the whole tuple
> count( for $x in dataset Tweets
> return $x
> )
> => Throws an exception Java heap exceeded. (The heap-size is less than the
> sum of AsterixDB configured memory ... so it's not a problem).
> {noformat}
> {noformat}
> 2- However, returning one field
> count( for $x in dataset Tweets
> return $x.id
> )
> => Worked just fine.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ASTERIXDB-1673) ARecordCaster potential object creation problem
[ https://issues.apache.org/jira/browse/ASTERIXDB-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wail Alkowaileet updated ASTERIXDB-1673: Priority: Trivial (was: Major) I changed it into Trivial because this should not be a problem currently. > ARecordCaster potential object creation problem > --- > > Key: ASTERIXDB-1673 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1673 > Project: Apache AsterixDB > Issue Type: Bug > Components: AsterixDB, Data Model >Reporter: Wail Alkowaileet >Assignee: Wail Alkowaileet >Priority: Trivial > > I use ACastVIsitor extensively in my work and I found that on > loadRequiredType: > The allocator will keep allocating Pointables (for typetag and fieldNames) > which would never be cleared. > I ran an experiment and got more than 6 million objects on a dataset of size > 1.5GB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ASTERIXDB-1673) ARecordCaster potential object creation problem
Wail Alkowaileet created ASTERIXDB-1673: --- Summary: ARecordCaster potential object creation problem Key: ASTERIXDB-1673 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1673 Project: Apache AsterixDB Issue Type: Bug Components: AsterixDB, Data Model Reporter: Wail Alkowaileet Assignee: Wail Alkowaileet I use ACastVIsitor extensively in my work and I found that on loadRequiredType: The allocator will keep allocating Pointables (for typetag and fieldNames) which would never be cleared. I ran an experiment and got more than 6 million objects on a dataset of size 1.5GB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (ASTERIXDB-1649) Read async query result more than once.
Wail Alkowaileet created ASTERIXDB-1649: --- Summary: Read async query result more than once. Key: ASTERIXDB-1649 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1649 Project: Apache AsterixDB Issue Type: Improvement Components: Hyracks Reporter: Wail Alkowaileet Currently, async query result does not allow reading the result files more than once even though the result file don't get deleted until the resultTTL is exceeded. It would good to read the materialized result files more than once. Specially for large queries which take so much time to materialize. The change would oneliner to support that. However, I want to know others feedback. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1414) Insert tweets with existed keys
[
https://issues.apache.org/jira/browse/ASTERIXDB-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489870#comment-15489870
]
Wail Alkowaileet commented on ASTERIXDB-1414:
-
Actually that feature is one of the best (among other things :-)).
I convinced a lot of people to use AsterixDB for Twitter just because this
feature as they suffered from duplicates when they use the API directly. This
can be way more expensive to clean and for storage size.
In a case my colleagues had, they collected tweets from the regular API and
found that each tweet repeated 30 time! Specially for highly selective
keywords. So the number of tweets are reduced by a factor of 30.
> Insert tweets with existed keys
> ---
>
> Key: ASTERIXDB-1414
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1414
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Feeds
>Reporter: Xikui Wang
>Assignee: Xikui Wang
>Priority: Minor
>
> When using the tweet id as key, some times there will have duplicate key
> exception that pop up in the terminal. Here is the error message.
> {quote}
> org.apache.asterix.common.exceptions.FrameDataException:
> org.apache.hyracks.storage.am.common.exceptions.TreeIndexDuplicateKeyException:
> Failed to insert key since key already exists.
> at
> org.apache.asterix.common.dataflow.AsterixLSMInsertDeleteOperatorNodePushable.nextFrame(AsterixLSMInsertDeleteOperatorNodePushable.java:120)
> at
> org.apache.asterix.external.feed.dataflow.FeedRuntimeInputHandler.process(FeedRuntimeInputHandler.java:265)
> at
> org.apache.asterix.external.feed.dataflow.FeedRuntimeInputHandler.nextFrame(FeedRuntimeInputHandler.java:127)
> at
> org.apache.asterix.external.operators.FeedMetaStoreNodePushable.nextFrame(FeedMetaStoreNodePushable.java:176)
> at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:349)
> at org.apache.hyracks.control.nc.Task.run(Task.java:297)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hyracks.storage.am.common.exceptions.TreeIndexDuplicateKeyException:
> Failed to insert key since key already exists.
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.insert(LSMBTree.java:360)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.modify(LSMBTree.java:333)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:353)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.forceModify(LSMHarness.java:333)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.forceInsert(LSMTreeIndexAccessor.java:158)
> at
> org.apache.asterix.common.dataflow.AsterixLSMInsertDeleteOperatorNodePushable.nextFrame(AsterixLSMInsertDeleteOperatorNodePushable.java:103)
> ... 8 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (ASTERIXDB-1641) Hit HYR002 exception when query a large dataset
[
https://issues.apache.org/jira/browse/ASTERIXDB-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489836#comment-15489836
]
Wail Alkowaileet edited comment on ASTERIXDB-1641 at 9/14/16 8:44 AM:
--
Probably then it's not the ResultSweepr problem?
It's 24hrs for someone who wants to run overnight job and come back later to
get the data.
was (Author: wyk):
Probably then it's not the ResultSweepr problem?
> Hit HYR002 exception when query a large dataset
> ---
>
> Key: ASTERIXDB-1641
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1641
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Environment: Master
>Reporter: Jianfeng Jia
>Assignee: Ian Maxon
>Priority: Blocker
>
> The twitter dataset now is above 100G. I hit an error when send a simple
> query to the dataset:
> {code}
> SEVERE: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0002:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDat
> aException: null
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:62)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataExceptio
> n: null
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:187)
> at
> org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:92)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:
> 69)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:125)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.close(AlgebricksMetaOperatorDescriptor.java:153)
> at
> org.apache.hyracks.dataflow.std.sort.AbstractExternalSortRunMerger.process(AbstractExternalSortRunMerger.java:167)
> at
> org.apache.hyracks.dataflow.std.sort.AbstractSorterOperatorDescriptor$MergeActivity$1.initialize(AbstractSorterOperatorDescriptor.java:194)
> at
>
[jira] [Commented] (ASTERIXDB-1641) Hit HYR002 exception when query a large dataset
[
https://issues.apache.org/jira/browse/ASTERIXDB-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489836#comment-15489836
]
Wail Alkowaileet commented on ASTERIXDB-1641:
-
Probably then it's not the ResultSweepr problem?
> Hit HYR002 exception when query a large dataset
> ---
>
> Key: ASTERIXDB-1641
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1641
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Environment: Master
>Reporter: Jianfeng Jia
>Assignee: Ian Maxon
>Priority: Blocker
>
> The twitter dataset now is above 100G. I hit an error when send a simple
> query to the dataset:
> {code}
> SEVERE: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0002:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDat
> aException: null
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:62)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataExceptio
> n: null
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:187)
> at
> org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:92)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:
> 69)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:125)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.close(AlgebricksMetaOperatorDescriptor.java:153)
> at
> org.apache.hyracks.dataflow.std.sort.AbstractExternalSortRunMerger.process(AbstractExternalSortRunMerger.java:167)
> at
> org.apache.hyracks.dataflow.std.sort.AbstractSorterOperatorDescriptor$MergeActivity$1.initialize(AbstractSorterOperatorDescriptor.java:194)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$1/1498483095.runAction(Unknown
> Source)
> at
>
[jira] [Comment Edited] (ASTERIXDB-1641) Hit HYR002 exception when query a large dataset
[
https://issues.apache.org/jira/browse/ASTERIXDB-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484913#comment-15484913
]
Wail Alkowaileet edited comment on ASTERIXDB-1641 at 9/12/16 6:41 PM:
--
It's in the cluster config file:
{noformat}8640{noformat}
If you look at
{noformat}org.apache.hyracks.control.common.dataset.ResultStateSweeper{noformat}
You will find that the result sweeper will destroy the result despite the fact
that the job is still running. It all depends on resultTTL (not
resultSweepThreshold. My bad).
In the documentation, it says that it only for async jobs, but I don't think
that's 100% true.
To reproduce:
1- Add breakpoint at line 64 in
{noformat}org.apache.hyracks.algebricks.runtime.serializer.ResultSerializerFactoryProvider{noformat}
2- Add breakpoint at line 75 in
{noformat}org.apache.hyracks.control.common.dataset.ResultStateSweeper{noformat}
3- Run any query you like.
4- From (1), execute line 64.
6- From (2), execute.
7- From (1), execute.
*Repeat one more time if that didn't produce an error.
You will see that your result file got deleted before writing.
was (Author: wyk):
If you look at
{noformat}org.apache.hyracks.control.common.dataset.ResultStateSweeper{noformat}
You will find that the result sweeper will destroy the result despite the fact
that the job is still running. It all depends on resultTTL (not
resultSweepThreshold. My bad).
In the documentation, it says that it only for async jobs, but I don't think
that's 100% true.
To reproduce:
1- Add breakpoint at line 64 in
{noformat}org.apache.hyracks.algebricks.runtime.serializer.ResultSerializerFactoryProvider{noformat}
2- Add breakpoint at line 75 in
{noformat}org.apache.hyracks.control.common.dataset.ResultStateSweeper{noformat}
3- Run any query you like.
4- From (1), execute line 64.
6- From (2), execute.
7- From (1), execute.
*Repeat one more time if that didn't produce an error.
You will see that your result file got deleted before writing.
> Hit HYR002 exception when query a large dataset
> ---
>
> Key: ASTERIXDB-1641
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1641
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Environment: Master
>Reporter: Jianfeng Jia
>Assignee: Ian Maxon
>Priority: Blocker
>
> The twitter dataset now is above 100G. I hit an error when send a simple
> query to the dataset:
> {code}
> SEVERE: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0002:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDat
> aException: null
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:62)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataExceptio
> n: null
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by:
[jira] [Resolved] (ASTERIXDB-1334) [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records' offset and length on nested ADM types
[
https://issues.apache.org/jira/browse/ASTERIXDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet resolved ASTERIXDB-1334.
-
Resolution: Fixed
> [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records'
> offset and length on nested ADM types
> ---
>
> Key: ASTERIXDB-1334
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1334
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Model
>Reporter: Wail Alkowaileet
>Assignee: Yingyi Bu
>Priority: Critical
>
> Hi AsterixDB team,
> There's a problem when I fully define the ADM type. The computed field length
> and offset seem to off.
> To reproduce:
> Download the data:
> https://www.dropbox.com/s/94llrvl08x85z9n/tweet.json?dl=0
> (Sorry it contains Arabic content)
> DDL:
> {noformat}
> create type coordinatesType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type geoType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type metadataType1 as open {
> 'iso_language_code':string,
> 'result_type':string
> }
> create type bounding_boxType1 as open {
> 'coordinates':[[[double]]],
> 'type':string
> }
> create type placeType1 as open {
> 'bounding_box':bounding_boxType1,
> 'country':string,
> 'country_code':string,
> 'full_name':string,
> 'id':string,
> 'name':string,
> 'place_type':string,
> 'url':string
> }
> create type listType1 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'indices':[int64],
> 'url':string
> }
> create type urlType1 as open {
> 'urls':[listType1]
> }
> create type listType2 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type descriptionType1 as open {
> 'urls':[listType2]
> }
> create type entitiesType1 as open {
> 'url':urlType1?,
> 'description':descriptionType1?
> }
> create type userType1 as open {
> 'contributors_enabled':boolean,
> 'created_at':string,
> 'default_profile':boolean,
> 'default_profile_image':boolean,
> 'description':string,
> 'favourites_count':int64,
> 'followers_count':int64,
> 'friends_count':int64,
> 'geo_enabled':boolean,
> 'has_extended_profile':boolean,
> 'id':int64,
> 'id_str':string,
> 'is_translation_enabled':boolean,
> 'is_translator':boolean,
> 'lang':string,
> 'listed_count':int64,
> 'location':string,
> 'name':string,
> 'profile_background_color':string,
> 'profile_background_image_url':string,
> 'profile_background_image_url_https':string,
> 'profile_background_tile':boolean,
> 'profile_banner_url':string?,
> 'profile_image_url':string,
> 'profile_image_url_https':string,
> 'profile_link_color':string,
> 'profile_sidebar_border_color':string,
> 'profile_sidebar_fill_color':string,
> 'profile_text_color':string,
> 'profile_use_background_image':boolean,
> 'protected':boolean,
> 'screen_name':string,
> 'statuses_count':int64,
> 'time_zone':string?,
> 'utc_offset':int64?,
> 'verified':boolean,
> 'entities':entitiesType1?,
> 'url':string?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type listType4 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType5 as open {
> 'id':int64?,
> 'id_str':string?,
> 'indices':[int64]?,
> 'name':string?,
> 'screen_name':string?
> }
> create type largeType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type mediumType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type smallType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type thumbType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type sizesType1 as open {
> 'large':largeType1,
> 'medium':mediumType1,
> 'small':smallType1,
> 'thumb':thumbType1
> }
> create type listType6 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'id':int64,
> 'id_str':string,
> 'indices':[int64],
> 'media_url':string,
> 'media_url_https':string,
> 'sizes':sizesType1,
> 'type':string,
> 'url':string
> }
> create type entitiesType2 as open {
> 'urls':[listType3],
> 'hashtags':[listType4]?,
> 'user_mentions':[listType5]?,
> 'media':[listType6]?
> }
> create type raw_sourceType1 as open {
> 'coordinates':coordinatesType1,
> 'created_at':string,
>
[jira] [Assigned] (ASTERIXDB-1627) create-polygon() from array of double is giving wrong numbers
[
https://issues.apache.org/jira/browse/ASTERIXDB-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet reassigned ASTERIXDB-1627:
---
Assignee: Wail Alkowaileet
> create-polygon() from array of double is giving wrong numbers
> -
>
> Key: ASTERIXDB-1627
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1627
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Formats, Data Model
>Reporter: Wail Alkowaileet
>Assignee: Wail Alkowaileet
>
> Hi,
> Data:
> https://www.dropbox.com/s/52c6uxufuldqzmm/statesJson.zip?dl=0
> DDL:
> {noformat}
> drop dataverse TwitterDataverse if exists
> create dataverse TwitterDataverse
> use dataverse TwitterDataverse
> create type StateType as {
> uid: uuid
> }
> create dataset States(StateType)
> primary key uid autogenerated
> {noformat}
> DML:
> {noformat}
> use dataverse TwitterDataverse
> load dataset States using localfs
> (("path"="localhost:///path/to/statesJson"),
> ("format"="adm"))
> {noformat}
> Get the array of doubles from the GeoJSON:
> {noformat}
> use dataverse TwitterDataverse
> let $polygon := (for $x in dataset States
> where $x.features[0].geometry.'type' = "Polygon"
> limit 1
> let $list := (for $i in $x.features[0].geometry.coordinates[0]
> for $j in $i
> for $k in $j
> return $k
> )
> return $list
> )
> return $polygon
> {noformat}
> Output:
> {noformat}
> [ -87.359296, 35.00118, -85.606675, 34.984749, -85.431413, 34.124869,
> -85.184951, 32.859696, -85.069935, 32.580372, -84.960397, 32.421541,
> -85.004212, 32.322956, -84.889196, 32.262709, -85.058981, 32.13674,
> -85.053504, 32.01077, -85.141136, 31.840985, -85.042551, 31.539753,
> -85.113751, 31.27686, -85.004212, 31.003013, -85.497137, 30.997536,
> -87.600282, 30.997536, -87.633143, 30.86609, -87.408589, 30.674397,
> -87.446927, 30.510088, -87.37025, 30.427934, -87.518128, 30.280057,
> -87.655051, 30.247195, -87.90699, 30.411504, -87.934375, 30.657966,
> -88.011052, 30.685351, -88.10416, 30.499135, -88.137022, 30.318396,
> -88.394438, 30.367688, -88.471115, 31.895754, -88.241084, 33.796253,
> -88.098683, 34.891641, -88.202745, 34.995703, -87.359296, 35.00118 ]
> {noformat}
> applying create-polygon() on the resulting array:
> {noformat}
> use dataverse TwitterDataverse
> let $polygon := (for $x in dataset States
> where $x.features[0].geometry.'type' = "Polygon"
> limit 1
> let $list := (for $i in $x.features[0].geometry.coordinates[0]
> for $j in $i
> for $k in $j
> return $k
> )
> return $list
> )
> return create-polygon($polygon)
> {noformat}
> Output:
> {noformat}
> polygon("2.920390972995509E-247,1.1352293178511514E-249
> 2.920084986948524E-247,1.1352270767181132E-249
> 2.9200543883612843E-247,1.1351097919833495E-249
> 2.920011359130714E-247,1.1349372266489597E-249
> 2.919991278753279E-247,1.1348991277964998E-249
> 2.919972154767195E-247,1.1348774637832571E-249
> 2.919979804326711E-247,1.1348640171214245E-249
> 2.919959723949276E-247,1.1348557996336175E-249
> 2.919989366319753E-247,1.1348386178864513E-249
> 2.91998841010299E-247,1.1348214360028882E-249
> 2.9200037095711975E-247,1.1347765887907304E-249
> 2.919986497844052E-247,1.134694414731042E-249
> 2.9199989284873825E-247,1.1346226992921991E-249
> 2.919979804326711E-247,1.1345479956759716E-249
> 2.92006586296244E-247,1.1345465015872795E-249
> 2.920433046183905E-247,1.1345465015872795E-249
> 2.920438783309895E-247,1.1345106440042545E-249
> 2.920399578946376E-247,1.134458351445616E-249
> 2.920406272289129E-247,1.134413529057645E-249
> 2.9203928854290354E-247,1.13439111856E-249
> 2.920418703107048E-247,1.1343507781509548E-249
> 2.9204426081769465E-247,1.134341813618802E-249
> 2.9204865936242805E-247,1.134386636006773E-249
> 2.920491374708095E-247,1.1344538691795396E-249
> 2.920504761568189E-247,1.1344613396230002E-249
> 2.920521017078572E-247,1.1344105411530538E-249
> 2.92052675437915E-247,1.1343612367717997E-249
> 2.9205716960432465E-247,1.1343746832972356E-249
> 2.92058508290334E-247,1.1347915294048584E-249
> 2.9205449223230584E-247,1.1350649698681716E-249
> 2.9205200608618086E-247,1.1352143771006265E-249
> 2.920538228805718E-247,1.1352285708068053E-249
> 2.920390972995509E-247,1.1352293178511514E-249")
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1641) Hit HYR002 exception when query a large dataset
[
https://issues.apache.org/jira/browse/ASTERIXDB-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484085#comment-15484085
]
Wail Alkowaileet commented on ASTERIXDB-1641:
-
Did you submit the query asynchronously or synchronously? The issue is due to
time constraints on the result files.
The solution is just increase the result sweep threshold. Async queries doesn't
have this issue.
> Hit HYR002 exception when query a large dataset
> ---
>
> Key: ASTERIXDB-1641
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1641
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Environment: Master
>Reporter: Jianfeng Jia
>Assignee: Ian Maxon
>Priority: Blocker
>
> The twitter dataset now is above 100G. I hit an error when send a simple
> query to the dataset:
> {code}
> SEVERE: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> HYR0002: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: HYR0002:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDat
> aException: null
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:62)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataExceptio
> n: null
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException: null
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:187)
> at
> org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:92)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:
> 69)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:125)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.close(AlgebricksMetaOperatorDescriptor.java:153)
> at
> org.apache.hyracks.dataflow.std.sort.AbstractExternalSortRunMerger.process(AbstractExternalSortRunMerger.java:167)
> at
> org.apache.hyracks.dataflow.std.sort.AbstractSorterOperatorDescriptor$MergeActivity$1.initialize(AbstractSorterOperatorDescriptor.java:194)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
>
[jira] [Commented] (ASTERIXDB-1334) [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records' offset and length on nested ADM types
[
https://issues.apache.org/jira/browse/ASTERIXDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484069#comment-15484069
]
Wail Alkowaileet commented on ASTERIXDB-1334:
-
I created synthetic data that reproduce the issue. I submitted it to gerrit.
> [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records'
> offset and length on nested ADM types
> ---
>
> Key: ASTERIXDB-1334
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1334
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Model
>Reporter: Wail Alkowaileet
>Assignee: Yingyi Bu
>Priority: Critical
>
> Hi AsterixDB team,
> There's a problem when I fully define the ADM type. The computed field length
> and offset seem to off.
> To reproduce:
> Download the data:
> https://www.dropbox.com/s/94llrvl08x85z9n/tweet.json?dl=0
> (Sorry it contains Arabic content)
> DDL:
> {noformat}
> create type coordinatesType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type geoType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type metadataType1 as open {
> 'iso_language_code':string,
> 'result_type':string
> }
> create type bounding_boxType1 as open {
> 'coordinates':[[[double]]],
> 'type':string
> }
> create type placeType1 as open {
> 'bounding_box':bounding_boxType1,
> 'country':string,
> 'country_code':string,
> 'full_name':string,
> 'id':string,
> 'name':string,
> 'place_type':string,
> 'url':string
> }
> create type listType1 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'indices':[int64],
> 'url':string
> }
> create type urlType1 as open {
> 'urls':[listType1]
> }
> create type listType2 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type descriptionType1 as open {
> 'urls':[listType2]
> }
> create type entitiesType1 as open {
> 'url':urlType1?,
> 'description':descriptionType1?
> }
> create type userType1 as open {
> 'contributors_enabled':boolean,
> 'created_at':string,
> 'default_profile':boolean,
> 'default_profile_image':boolean,
> 'description':string,
> 'favourites_count':int64,
> 'followers_count':int64,
> 'friends_count':int64,
> 'geo_enabled':boolean,
> 'has_extended_profile':boolean,
> 'id':int64,
> 'id_str':string,
> 'is_translation_enabled':boolean,
> 'is_translator':boolean,
> 'lang':string,
> 'listed_count':int64,
> 'location':string,
> 'name':string,
> 'profile_background_color':string,
> 'profile_background_image_url':string,
> 'profile_background_image_url_https':string,
> 'profile_background_tile':boolean,
> 'profile_banner_url':string?,
> 'profile_image_url':string,
> 'profile_image_url_https':string,
> 'profile_link_color':string,
> 'profile_sidebar_border_color':string,
> 'profile_sidebar_fill_color':string,
> 'profile_text_color':string,
> 'profile_use_background_image':boolean,
> 'protected':boolean,
> 'screen_name':string,
> 'statuses_count':int64,
> 'time_zone':string?,
> 'utc_offset':int64?,
> 'verified':boolean,
> 'entities':entitiesType1?,
> 'url':string?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type listType4 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType5 as open {
> 'id':int64?,
> 'id_str':string?,
> 'indices':[int64]?,
> 'name':string?,
> 'screen_name':string?
> }
> create type largeType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type mediumType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type smallType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type thumbType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type sizesType1 as open {
> 'large':largeType1,
> 'medium':mediumType1,
> 'small':smallType1,
> 'thumb':thumbType1
> }
> create type listType6 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'id':int64,
> 'id_str':string,
> 'indices':[int64],
> 'media_url':string,
> 'media_url_https':string,
> 'sizes':sizesType1,
> 'type':string,
> 'url':string
> }
> create type entitiesType2 as open {
> 'urls':[listType3],
> 'hashtags':[listType4]?,
> 'user_mentions':[listType5]?,
> 'media':[listType6]?
> }
> create type raw_sourceType1 as open
[jira] [Commented] (ASTERIXDB-1334) [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records' offset and length on nested ADM types
[
https://issues.apache.org/jira/browse/ASTERIXDB-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483126#comment-15483126
]
Wail Alkowaileet commented on ASTERIXDB-1334:
-
I wanted to add the same dataset ... but I wasn't sure if that is ok
legalwise/copyright ?
> [IndexOutOfBoundsException/ArrayIndexOutOfBoundsException] Malformed records'
> offset and length on nested ADM types
> ---
>
> Key: ASTERIXDB-1334
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1334
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Model
>Reporter: Wail Alkowaileet
>Assignee: Yingyi Bu
>Priority: Critical
>
> Hi AsterixDB team,
> There's a problem when I fully define the ADM type. The computed field length
> and offset seem to off.
> To reproduce:
> Download the data:
> https://www.dropbox.com/s/94llrvl08x85z9n/tweet.json?dl=0
> (Sorry it contains Arabic content)
> DDL:
> {noformat}
> create type coordinatesType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type geoType1 as open {
> 'coordinates':[double],
> 'type':string
> }
> create type metadataType1 as open {
> 'iso_language_code':string,
> 'result_type':string
> }
> create type bounding_boxType1 as open {
> 'coordinates':[[[double]]],
> 'type':string
> }
> create type placeType1 as open {
> 'bounding_box':bounding_boxType1,
> 'country':string,
> 'country_code':string,
> 'full_name':string,
> 'id':string,
> 'name':string,
> 'place_type':string,
> 'url':string
> }
> create type listType1 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'indices':[int64],
> 'url':string
> }
> create type urlType1 as open {
> 'urls':[listType1]
> }
> create type listType2 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type descriptionType1 as open {
> 'urls':[listType2]
> }
> create type entitiesType1 as open {
> 'url':urlType1?,
> 'description':descriptionType1?
> }
> create type userType1 as open {
> 'contributors_enabled':boolean,
> 'created_at':string,
> 'default_profile':boolean,
> 'default_profile_image':boolean,
> 'description':string,
> 'favourites_count':int64,
> 'followers_count':int64,
> 'friends_count':int64,
> 'geo_enabled':boolean,
> 'has_extended_profile':boolean,
> 'id':int64,
> 'id_str':string,
> 'is_translation_enabled':boolean,
> 'is_translator':boolean,
> 'lang':string,
> 'listed_count':int64,
> 'location':string,
> 'name':string,
> 'profile_background_color':string,
> 'profile_background_image_url':string,
> 'profile_background_image_url_https':string,
> 'profile_background_tile':boolean,
> 'profile_banner_url':string?,
> 'profile_image_url':string,
> 'profile_image_url_https':string,
> 'profile_link_color':string,
> 'profile_sidebar_border_color':string,
> 'profile_sidebar_fill_color':string,
> 'profile_text_color':string,
> 'profile_use_background_image':boolean,
> 'protected':boolean,
> 'screen_name':string,
> 'statuses_count':int64,
> 'time_zone':string?,
> 'utc_offset':int64?,
> 'verified':boolean,
> 'entities':entitiesType1?,
> 'url':string?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'expanded_url':string?,
> 'indices':[int64]?,
> 'url':string?
> }
> create type listType4 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType5 as open {
> 'id':int64?,
> 'id_str':string?,
> 'indices':[int64]?,
> 'name':string?,
> 'screen_name':string?
> }
> create type largeType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type mediumType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type smallType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type thumbType1 as open {
> 'h':int64,
> 'resize':string,
> 'w':int64
> }
> create type sizesType1 as open {
> 'large':largeType1,
> 'medium':mediumType1,
> 'small':smallType1,
> 'thumb':thumbType1
> }
> create type listType6 as open {
> 'display_url':string,
> 'expanded_url':string,
> 'id':int64,
> 'id_str':string,
> 'indices':[int64],
> 'media_url':string,
> 'media_url_https':string,
> 'sizes':sizesType1,
> 'type':string,
> 'url':string
> }
> create type entitiesType2 as open {
> 'urls':[listType3],
> 'hashtags':[listType4]?,
> 'user_mentions':[listType5]?,
> 'media':[listType6]?
> }
> create type
[jira] [Comment Edited] (ASTERIXDB-1616) NPE when printing record inside open type with unicode fields
[ https://issues.apache.org/jira/browse/ASTERIXDB-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482171#comment-15482171 ] Wail Alkowaileet edited comment on ASTERIXDB-1616 at 9/11/16 6:30 PM: -- Hi Ian, I apologies for hijacking this ... but after 7 hours of debugging I found the bug! it's just one line that causes the issue. I'm submitting the patch after some little testing. was (Author: wyk): Hi Ian, I apologies for hijacking this ... but after 7 hours of debugging I found the bug! > NPE when printing record inside open type with unicode fields > - > > Key: ASTERIXDB-1616 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1616 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Ian Maxon >Assignee: Ian Maxon > > DDL: > https://github.com/kevincoakley/asterixdb_tests/blob/master/notebooks/asterixdb-spark/Count%20one_percent%20Tweets%20Spark%20Single.ipynb > Data: > https://object.cloud.sdsc.edu/v1/AUTH_kcoakley/asterixdblogs/2015_11_07_00_onepercent.txt > Basically just a scan+limit on the one_percent dataset will give > IndexOutOfBounds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1616) NPE when printing record inside open type with unicode fields
[ https://issues.apache.org/jira/browse/ASTERIXDB-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482171#comment-15482171 ] Wail Alkowaileet commented on ASTERIXDB-1616: - Hi Ian, I apologies for hijacking this ... but after 7 hours of debugging I found the bug! > NPE when printing record inside open type with unicode fields > - > > Key: ASTERIXDB-1616 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1616 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Ian Maxon >Assignee: Ian Maxon > > DDL: > https://github.com/kevincoakley/asterixdb_tests/blob/master/notebooks/asterixdb-spark/Count%20one_percent%20Tweets%20Spark%20Single.ipynb > Data: > https://object.cloud.sdsc.edu/v1/AUTH_kcoakley/asterixdblogs/2015_11_07_00_onepercent.txt > Basically just a scan+limit on the one_percent dataset will give > IndexOutOfBounds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1627) create-polygon() from array of double is giving wrong numbers
[
https://issues.apache.org/jira/browse/ASTERIXDB-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457874#comment-15457874
]
Wail Alkowaileet commented on ASTERIXDB-1627:
-
I suspect this bug is related to:
https://issues.apache.org/jira/browse/ASTERIXDB-1616
and
https://issues.apache.org/jira/browse/ASTERIXDB-1334
No sure though.
> create-polygon() from array of double is giving wrong numbers
> -
>
> Key: ASTERIXDB-1627
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1627
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Formats, Data Model
>Reporter: Wail Alkowaileet
>
> Hi,
> Data:
> https://www.dropbox.com/s/52c6uxufuldqzmm/statesJson.zip?dl=0
> DDL:
> {noformat}
> drop dataverse TwitterDataverse if exists
> create dataverse TwitterDataverse
> use dataverse TwitterDataverse
> create type StateType as {
> uid: uuid
> }
> create dataset States(StateType)
> primary key uid autogenerated
> {noformat}
> DML:
> {noformat}
> use dataverse TwitterDataverse
> load dataset States using localfs
> (("path"="localhost:///path/to/statesJson"),
> ("format"="adm"))
> {noformat}
> Get the array of doubles from the GeoJSON:
> {noformat}
> use dataverse TwitterDataverse
> let $polygon := (for $x in dataset States
> where $x.features[0].geometry.'type' = "Polygon"
> limit 1
> let $list := (for $i in $x.features[0].geometry.coordinates[0]
> for $j in $i
> for $k in $j
> return $k
> )
> return $list
> )
> return $polygon
> {noformat}
> Output:
> {noformat}
> [ -87.359296, 35.00118, -85.606675, 34.984749, -85.431413, 34.124869,
> -85.184951, 32.859696, -85.069935, 32.580372, -84.960397, 32.421541,
> -85.004212, 32.322956, -84.889196, 32.262709, -85.058981, 32.13674,
> -85.053504, 32.01077, -85.141136, 31.840985, -85.042551, 31.539753,
> -85.113751, 31.27686, -85.004212, 31.003013, -85.497137, 30.997536,
> -87.600282, 30.997536, -87.633143, 30.86609, -87.408589, 30.674397,
> -87.446927, 30.510088, -87.37025, 30.427934, -87.518128, 30.280057,
> -87.655051, 30.247195, -87.90699, 30.411504, -87.934375, 30.657966,
> -88.011052, 30.685351, -88.10416, 30.499135, -88.137022, 30.318396,
> -88.394438, 30.367688, -88.471115, 31.895754, -88.241084, 33.796253,
> -88.098683, 34.891641, -88.202745, 34.995703, -87.359296, 35.00118 ]
> {noformat}
> applying create-polygon() on the resulting array:
> {noformat}
> use dataverse TwitterDataverse
> let $polygon := (for $x in dataset States
> where $x.features[0].geometry.'type' = "Polygon"
> limit 1
> let $list := (for $i in $x.features[0].geometry.coordinates[0]
> for $j in $i
> for $k in $j
> return $k
> )
> return $list
> )
> return create-polygon($polygon)
> {noformat}
> Output:
> {noformat}
> polygon("2.920390972995509E-247,1.1352293178511514E-249
> 2.920084986948524E-247,1.1352270767181132E-249
> 2.9200543883612843E-247,1.1351097919833495E-249
> 2.920011359130714E-247,1.1349372266489597E-249
> 2.919991278753279E-247,1.1348991277964998E-249
> 2.919972154767195E-247,1.1348774637832571E-249
> 2.919979804326711E-247,1.1348640171214245E-249
> 2.919959723949276E-247,1.1348557996336175E-249
> 2.919989366319753E-247,1.1348386178864513E-249
> 2.91998841010299E-247,1.1348214360028882E-249
> 2.9200037095711975E-247,1.1347765887907304E-249
> 2.919986497844052E-247,1.134694414731042E-249
> 2.9199989284873825E-247,1.1346226992921991E-249
> 2.919979804326711E-247,1.1345479956759716E-249
> 2.92006586296244E-247,1.1345465015872795E-249
> 2.920433046183905E-247,1.1345465015872795E-249
> 2.920438783309895E-247,1.1345106440042545E-249
> 2.920399578946376E-247,1.134458351445616E-249
> 2.920406272289129E-247,1.134413529057645E-249
> 2.9203928854290354E-247,1.13439111856E-249
> 2.920418703107048E-247,1.1343507781509548E-249
> 2.9204426081769465E-247,1.134341813618802E-249
> 2.9204865936242805E-247,1.134386636006773E-249
> 2.920491374708095E-247,1.1344538691795396E-249
> 2.920504761568189E-247,1.1344613396230002E-249
> 2.920521017078572E-247,1.1344105411530538E-249
> 2.92052675437915E-247,1.1343612367717997E-249
> 2.9205716960432465E-247,1.1343746832972356E-249
> 2.92058508290334E-247,1.1347915294048584E-249
> 2.9205449223230584E-247,1.1350649698681716E-249
> 2.9205200608618086E-247,1.1352143771006265E-249
> 2.920538228805718E-247,1.1352285708068053E-249
> 2.920390972995509E-247,1.1352293178511514E-249")
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ASTERIXDB-1627) create-polygon() from array of double is giving wrong numbers
[
https://issues.apache.org/jira/browse/ASTERIXDB-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-1627:
Description:
Hi,
Data:
https://www.dropbox.com/s/52c6uxufuldqzmm/statesJson.zip?dl=0
DDL:
{noformat}
drop dataverse TwitterDataverse if exists
create dataverse TwitterDataverse
use dataverse TwitterDataverse
create type StateType as {
uid: uuid
}
create dataset States(StateType)
primary key uid autogenerated
{noformat}
DML:
{noformat}
use dataverse TwitterDataverse
load dataset States using localfs
(("path"="localhost:///path/to/statesJson"),
("format"="adm"))
{noformat}
Get the array of doubles from the GeoJSON:
{noformat}
use dataverse TwitterDataverse
let $polygon := (for $x in dataset States
where $x.features[0].geometry.'type' = "Polygon"
limit 1
let $list := (for $i in $x.features[0].geometry.coordinates[0]
for $j in $i
for $k in $j
return $k
)
return $list
)
return $polygon
{noformat}
Output:
{noformat}
[ -87.359296, 35.00118, -85.606675, 34.984749, -85.431413, 34.124869,
-85.184951, 32.859696, -85.069935, 32.580372, -84.960397, 32.421541,
-85.004212, 32.322956, -84.889196, 32.262709, -85.058981, 32.13674, -85.053504,
32.01077, -85.141136, 31.840985, -85.042551, 31.539753, -85.113751, 31.27686,
-85.004212, 31.003013, -85.497137, 30.997536, -87.600282, 30.997536,
-87.633143, 30.86609, -87.408589, 30.674397, -87.446927, 30.510088, -87.37025,
30.427934, -87.518128, 30.280057, -87.655051, 30.247195, -87.90699, 30.411504,
-87.934375, 30.657966, -88.011052, 30.685351, -88.10416, 30.499135, -88.137022,
30.318396, -88.394438, 30.367688, -88.471115, 31.895754, -88.241084, 33.796253,
-88.098683, 34.891641, -88.202745, 34.995703, -87.359296, 35.00118 ]
{noformat}
applying create-polygon() on the resulting array:
{noformat}
use dataverse TwitterDataverse
let $polygon := (for $x in dataset States
where $x.features[0].geometry.'type' = "Polygon"
limit 1
let $list := (for $i in $x.features[0].geometry.coordinates[0]
for $j in $i
for $k in $j
return $k
)
return $list
)
return create-polygon($polygon)
{noformat}
Output:
{noformat}
polygon("2.920390972995509E-247,1.1352293178511514E-249
2.920084986948524E-247,1.1352270767181132E-249
2.9200543883612843E-247,1.1351097919833495E-249
2.920011359130714E-247,1.1349372266489597E-249
2.919991278753279E-247,1.1348991277964998E-249
2.919972154767195E-247,1.1348774637832571E-249
2.919979804326711E-247,1.1348640171214245E-249
2.919959723949276E-247,1.1348557996336175E-249
2.919989366319753E-247,1.1348386178864513E-249
2.91998841010299E-247,1.1348214360028882E-249
2.9200037095711975E-247,1.1347765887907304E-249
2.919986497844052E-247,1.134694414731042E-249
2.9199989284873825E-247,1.1346226992921991E-249
2.919979804326711E-247,1.1345479956759716E-249
2.92006586296244E-247,1.1345465015872795E-249
2.920433046183905E-247,1.1345465015872795E-249
2.920438783309895E-247,1.1345106440042545E-249
2.920399578946376E-247,1.134458351445616E-249
2.920406272289129E-247,1.134413529057645E-249
2.9203928854290354E-247,1.13439111856E-249
2.920418703107048E-247,1.1343507781509548E-249
2.9204426081769465E-247,1.134341813618802E-249
2.9204865936242805E-247,1.134386636006773E-249
2.920491374708095E-247,1.1344538691795396E-249
2.920504761568189E-247,1.1344613396230002E-249
2.920521017078572E-247,1.1344105411530538E-249
2.92052675437915E-247,1.1343612367717997E-249
2.9205716960432465E-247,1.1343746832972356E-249
2.92058508290334E-247,1.1347915294048584E-249
2.9205449223230584E-247,1.1350649698681716E-249
2.9205200608618086E-247,1.1352143771006265E-249
2.920538228805718E-247,1.1352285708068053E-249
2.920390972995509E-247,1.1352293178511514E-249")
{noformat}
was:
Hi,
Data:
https://www.dropbox.com/s/52c6uxufuldqzmm/statesJson.zip?dl=0
DDL:
{noformat}
drop dataverse TwitterDataverse if exists
create dataverse TwitterDataverse
use dataverse TwitterDataverse
create type StateType as {
uid: uuid
}
create dataset States(StateType)
primary key uid autogenerated
{noformat}
DML:
{noformat}
use dataverse TwitterDataverse
load dataset States using localfs
(("path"="localhost:///path/to/statesJson"),
("format"="adm"))
{noformat}
Get the array of doubles from the GeoJSON:
{noformat}
use dataverse TwitterDataverse
let $polygon := (for $x in dataset States
where $x.features[0].geometry.'type' = "Polygon"
limit 1
let $list := (for $i in $x.features[0].geometry.coordinates[0]
for $j in $i
for $k in $j
return $k
)
return $list
)
return $polygon
{noformat}
Output:
{noformat}
[ -87.359296, 35.00118, -85.606675, 34.984749, -85.431413, 34.124869,
-85.184951, 32.859696, -85.069935, 32.580372, -84.960397, 32.421541,
-85.004212, 32.322956, -84.889196, 32.262709, -85.058981, 32.13674, -85.053504,
32.01077, -85.141136, 31.840985, -85.042551, 31.539753, -85.113751, 31.27686,
-85.004212, 31.003013, -85.497137, 30.997536, -87.600282, 30.997536,
[jira] [Created] (ASTERIXDB-1627) create-polygon() from array of double is giving wrong numbers
Wail Alkowaileet created ASTERIXDB-1627:
---
Summary: create-polygon() from array of double is giving wrong
numbers
Key: ASTERIXDB-1627
URL: https://issues.apache.org/jira/browse/ASTERIXDB-1627
Project: Apache AsterixDB
Issue Type: Bug
Components: Data Formats, Data Model
Reporter: Wail Alkowaileet
Hi,
Data:
https://www.dropbox.com/s/52c6uxufuldqzmm/statesJson.zip?dl=0
DDL:
{noformat}
drop dataverse TwitterDataverse if exists
create dataverse TwitterDataverse
use dataverse TwitterDataverse
create type StateType as {
uid: uuid
}
create dataset States(StateType)
primary key uid autogenerated
{noformat}
DML:
{noformat}
use dataverse TwitterDataverse
load dataset States using localfs
(("path"="localhost:///path/to/statesJson"),
("format"="adm"))
{noformat}
Get the array of doubles from the GeoJSON:
{noformat}
use dataverse TwitterDataverse
let $polygon := (for $x in dataset States
where $x.features[0].geometry.'type' = "Polygon"
limit 1
let $list := (for $i in $x.features[0].geometry.coordinates[0]
for $j in $i
for $k in $j
return $k
)
return $list
)
return $polygon
{noformat}
Output:
{noformat}
[ -87.359296, 35.00118, -85.606675, 34.984749, -85.431413, 34.124869,
-85.184951, 32.859696, -85.069935, 32.580372, -84.960397, 32.421541,
-85.004212, 32.322956, -84.889196, 32.262709, -85.058981, 32.13674, -85.053504,
32.01077, -85.141136, 31.840985, -85.042551, 31.539753, -85.113751, 31.27686,
-85.004212, 31.003013, -85.497137, 30.997536, -87.600282, 30.997536,
-87.633143, 30.86609, -87.408589, 30.674397, -87.446927, 30.510088, -87.37025,
30.427934, -87.518128, 30.280057, -87.655051, 30.247195, -87.90699, 30.411504,
-87.934375, 30.657966, -88.011052, 30.685351, -88.10416, 30.499135, -88.137022,
30.318396, -88.394438, 30.367688, -88.471115, 31.895754, -88.241084, 33.796253,
-88.098683, 34.891641, -88.202745, 34.995703, -87.359296, 35.00118 ]
{noformat}
applying create-polygon() on the resulting array:
{noformat}
use dataverse TwitterDataverse
let $polygon := (for $x in dataset States
where $x.features[0].geometry.'type' = "Polygon"
limit 1
let $list := (for $i in $x.features[0].geometry.coordinates[0]
for $j in $i
for $k in $j
return $k
)
return $list
)
return create-polygon($polygon)
Output:
{noformat}
polygon("2.920390972995509E-247,1.1352293178511514E-249
2.920084986948524E-247,1.1352270767181132E-249
2.9200543883612843E-247,1.1351097919833495E-249
2.920011359130714E-247,1.1349372266489597E-249
2.919991278753279E-247,1.1348991277964998E-249
2.919972154767195E-247,1.1348774637832571E-249
2.919979804326711E-247,1.1348640171214245E-249
2.919959723949276E-247,1.1348557996336175E-249
2.919989366319753E-247,1.1348386178864513E-249
2.91998841010299E-247,1.1348214360028882E-249
2.9200037095711975E-247,1.1347765887907304E-249
2.919986497844052E-247,1.134694414731042E-249
2.9199989284873825E-247,1.1346226992921991E-249
2.919979804326711E-247,1.1345479956759716E-249
2.92006586296244E-247,1.1345465015872795E-249
2.920433046183905E-247,1.1345465015872795E-249
2.920438783309895E-247,1.1345106440042545E-249
2.920399578946376E-247,1.134458351445616E-249
2.920406272289129E-247,1.134413529057645E-249
2.9203928854290354E-247,1.13439111856E-249
2.920418703107048E-247,1.1343507781509548E-249
2.9204426081769465E-247,1.134341813618802E-249
2.9204865936242805E-247,1.134386636006773E-249
2.920491374708095E-247,1.1344538691795396E-249
2.920504761568189E-247,1.1344613396230002E-249
2.920521017078572E-247,1.1344105411530538E-249
2.92052675437915E-247,1.1343612367717997E-249
2.9205716960432465E-247,1.1343746832972356E-249
2.92058508290334E-247,1.1347915294048584E-249
2.9205449223230584E-247,1.1350649698681716E-249
2.9205200608618086E-247,1.1352143771006265E-249
2.920538228805718E-247,1.1352285708068053E-249
2.920390972995509E-247,1.1352293178511514E-249")
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1616) NPE when printing record inside open type with unicode fields
[ https://issues.apache.org/jira/browse/ASTERIXDB-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457826#comment-15457826 ] Wail Alkowaileet commented on ASTERIXDB-1616: - Hi Ian, Unfortunately, I tried to see what's going on but really couldn't figure it out. > NPE when printing record inside open type with unicode fields > - > > Key: ASTERIXDB-1616 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1616 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Ian Maxon >Assignee: Ian Maxon > > DDL: > https://github.com/kevincoakley/asterixdb_tests/blob/master/notebooks/asterixdb-spark/Count%20one_percent%20Tweets%20Spark%20Single.ipynb > Data: > https://object.cloud.sdsc.edu/v1/AUTH_kcoakley/asterixdblogs/2015_11_07_00_onepercent.txt > Basically just a scan+limit on the one_percent dataset will give > IndexOutOfBounds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1616) NPE when printing record inside open type with unicode fields
[ https://issues.apache.org/jira/browse/ASTERIXDB-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451212#comment-15451212 ] Wail Alkowaileet commented on ASTERIXDB-1616: - The problem occurs with variable length fields. When you project (return) only primitive types at level zero (the dataset type), the issue disappears. I remember I tried to debug it and it seems that the length offset of a string (for example) is off by one byte (which leads to read some random 4-bytes that are way larger than the frame size. Hence, it throws IndexOutOfBoundExcpetion sometimes). However, if you make the schema all open (key-only) this should solve the problem. Which makes me wonder if that has anything to do with the parser? I suspect the problem resides in the ARecordBuilder and friends. I will try to print the record right before flush (and after the parsing) and see if I get the same issue. > NPE when printing record inside open type with unicode fields > - > > Key: ASTERIXDB-1616 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1616 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Ian Maxon >Assignee: Ian Maxon > > DDL: > https://github.com/kevincoakley/asterixdb_tests/blob/master/notebooks/asterixdb-spark/Count%20one_percent%20Tweets%20Spark%20Single.ipynb > Data: > https://object.cloud.sdsc.edu/v1/AUTH_kcoakley/asterixdblogs/2015_11_07_00_onepercent.txt > Basically just a scan+limit on the one_percent dataset will give > IndexOutOfBounds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (ASTERIXDB-1616) NPE when printing record inside open type with unicode fields
[ https://issues.apache.org/jira/browse/ASTERIXDB-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449724#comment-15449724 ] Wail Alkowaileet edited comment on ASTERIXDB-1616 at 8/30/16 6:18 PM: -- Does the error appear if everything is open field (i.e key-only type) ? If not, I think this should be the same problem as https://issues.apache.org/jira/browse/ASTERIXDB-1334 >From Yingyi: It seems that the parser has some issues with unicode?. was (Author: wyk): Does the error appear if everything is open field (i.e key-only type) ? If not, I think this should be the same problem as https://issues.apache.org/jira/browse/ASTERIXDB-1334 > NPE when printing record inside open type with unicode fields > - > > Key: ASTERIXDB-1616 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1616 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Ian Maxon >Assignee: Ian Maxon > > DDL: > https://github.com/kevincoakley/asterixdb_tests/blob/master/notebooks/asterixdb-spark/Count%20one_percent%20Tweets%20Spark%20Single.ipynb > Data: > https://object.cloud.sdsc.edu/v1/AUTH_kcoakley/asterixdblogs/2015_11_07_00_onepercent.txt > Basically just a scan+limit on the one_percent dataset will give > IndexOutOfBounds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ASTERIXDB-1609) TwitterParser does not parse closed-nullable fields.
[
https://issues.apache.org/jira/browse/ASTERIXDB-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wail Alkowaileet updated ASTERIXDB-1609:
Summary: TwitterParser does not parse closed-nullable fields. (was:
TwitterParser does parse closed-nullable fields.)
> TwitterParser does not parse closed-nullable fields.
>
>
> Key: ASTERIXDB-1609
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1609
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Model, Feeds
>Reporter: Wail Alkowaileet
>Assignee: Xikui Wang
>
> TwitterParser doesn't support types NULL/MISSING
> To reproduce, connect TwitterFeed using the defined type below. I
> roundtrip-ed it using load statement and it was successfully loaded.
> DDL:
> {noformat}
> drop dataverse feeds if exists;
> create dataverse feeds;
> use dataverse feeds;
> create type userType1 as open {
> 'utc_offset':int64?,
> 'friends_count':int64?,
> 'profile_image_url_https':string?,
> 'listed_count':int64?,
> 'profile_background_image_url':string?,
> 'default_profile_image':boolean?,
> 'favourites_count':int64?,
> 'description':string?,
> 'created_at':string?,
> 'is_translator':boolean?,
> 'profile_background_image_url_https':string?,
> 'protected':boolean?,
> 'screen_name':string?,
> 'id_str':string?,
> 'profile_link_color':string?,
> 'id':int64?,
> 'geo_enabled':boolean?,
> 'profile_background_color':string?,
> 'lang':string?,
> 'profile_sidebar_border_color':string?,
> 'profile_text_color':string?,
> 'verified':boolean?,
> 'profile_image_url':string?,
> 'time_zone':string?,
> 'contributors_enabled':boolean?,
> 'profile_background_tile':boolean?,
> 'profile_banner_url':string?,
> 'statuses_count':int64?,
> 'followers_count':int64?,
> 'profile_use_background_image':boolean?,
> 'default_profile':boolean?,
> 'name':string?,
> 'location':string?,
> 'profile_sidebar_fill_color':string?,
> 'url':string?
> }
> create type smallType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type largeType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type thumbType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type mediumType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type sizesType1 as open {
> 'small':smallType1?,
> 'large':largeType1?,
> 'thumb':thumbType1?,
> 'medium':mediumType1?
> }
> create type listType2 as open {
> 'content_type':string?,
> 'bitrate':int64?,
> 'url':string?
> }
> create type video_infoType1 as open {
> 'aspect_ratio':[int64]?,
> 'duration_millis':int64?,
> 'variants':[listType2]?
> }
> create type listType1 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'sizes':sizesType1?,
> 'id_str':string?,
> 'expanded_url':string?,
> 'media_url_https':string?,
> 'id':int64?,
> 'type':string?,
> 'media_url':string?,
> 'url':string?,
> 'video_info':video_infoType1?,
> 'source_user_id':int64?,
> 'source_status_id':int64?,
> 'source_status_id_str':string?,
> 'source_user_id_str':string?
> }
> create type extended_entitiesType1 as open {
> 'media':[listType1]?
> }
> create type smallType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type largeType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type thumbType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type mediumType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type sizesType2 as open {
> 'small':smallType2?,
> 'large':largeType2?,
> 'thumb':thumbType2?,
> 'medium':mediumType2?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'sizes':sizesType2?,
> 'id_str':string?,
> 'expanded_url':string?,
> 'media_url_https':string?,
> 'id':int64?,
> 'type':string?,
> 'media_url':string?,
> 'url':string?,
> 'source_user_id':int64?,
> 'source_status_id':int64?,
> 'source_status_id_str':string?,
> 'source_user_id_str':string?
> }
> create type listType4 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'expanded_url':string?,
> 'url':string?
> }
> create type listType5 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType6 as open {
> 'indices':[int64]?,
> 'screen_name':string?,
> 'id_str':string?,
> 'name':string?,
> 'id':int64?
> }
> create type entitiesType1 as open {
>
[jira] [Commented] (ASTERIXDB-1609) TwitterParser does parse closed-nullable fields.
[
https://issues.apache.org/jira/browse/ASTERIXDB-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439529#comment-15439529
]
Wail Alkowaileet commented on ASTERIXDB-1609:
-
I have another question:
Why parsing the Tweets using JSONObject ? why not using the ADMParser instead?
> TwitterParser does parse closed-nullable fields.
>
>
> Key: ASTERIXDB-1609
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1609
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Model, Feeds
>Reporter: Wail Alkowaileet
>
> TwitterParser doesn't support types NULL/MISSING
> To reproduce, connect TwitterFeed using the defined type below. I
> roundtrip-ed it using load statement and it was successfully loaded.
> DDL:
> {noformat}
> drop dataverse feeds if exists;
> create dataverse feeds;
> use dataverse feeds;
> create type userType1 as open {
> 'utc_offset':int64?,
> 'friends_count':int64?,
> 'profile_image_url_https':string?,
> 'listed_count':int64?,
> 'profile_background_image_url':string?,
> 'default_profile_image':boolean?,
> 'favourites_count':int64?,
> 'description':string?,
> 'created_at':string?,
> 'is_translator':boolean?,
> 'profile_background_image_url_https':string?,
> 'protected':boolean?,
> 'screen_name':string?,
> 'id_str':string?,
> 'profile_link_color':string?,
> 'id':int64?,
> 'geo_enabled':boolean?,
> 'profile_background_color':string?,
> 'lang':string?,
> 'profile_sidebar_border_color':string?,
> 'profile_text_color':string?,
> 'verified':boolean?,
> 'profile_image_url':string?,
> 'time_zone':string?,
> 'contributors_enabled':boolean?,
> 'profile_background_tile':boolean?,
> 'profile_banner_url':string?,
> 'statuses_count':int64?,
> 'followers_count':int64?,
> 'profile_use_background_image':boolean?,
> 'default_profile':boolean?,
> 'name':string?,
> 'location':string?,
> 'profile_sidebar_fill_color':string?,
> 'url':string?
> }
> create type smallType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type largeType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type thumbType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type mediumType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type sizesType1 as open {
> 'small':smallType1?,
> 'large':largeType1?,
> 'thumb':thumbType1?,
> 'medium':mediumType1?
> }
> create type listType2 as open {
> 'content_type':string?,
> 'bitrate':int64?,
> 'url':string?
> }
> create type video_infoType1 as open {
> 'aspect_ratio':[int64]?,
> 'duration_millis':int64?,
> 'variants':[listType2]?
> }
> create type listType1 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'sizes':sizesType1?,
> 'id_str':string?,
> 'expanded_url':string?,
> 'media_url_https':string?,
> 'id':int64?,
> 'type':string?,
> 'media_url':string?,
> 'url':string?,
> 'video_info':video_infoType1?,
> 'source_user_id':int64?,
> 'source_status_id':int64?,
> 'source_status_id_str':string?,
> 'source_user_id_str':string?
> }
> create type extended_entitiesType1 as open {
> 'media':[listType1]?
> }
> create type smallType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type largeType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type thumbType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type mediumType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type sizesType2 as open {
> 'small':smallType2?,
> 'large':largeType2?,
> 'thumb':thumbType2?,
> 'medium':mediumType2?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'sizes':sizesType2?,
> 'id_str':string?,
> 'expanded_url':string?,
> 'media_url_https':string?,
> 'id':int64?,
> 'type':string?,
> 'media_url':string?,
> 'url':string?,
> 'source_user_id':int64?,
> 'source_status_id':int64?,
> 'source_status_id_str':string?,
> 'source_user_id_str':string?
> }
> create type listType4 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'expanded_url':string?,
> 'url':string?
> }
> create type listType5 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType6 as open {
> 'indices':[int64]?,
> 'screen_name':string?,
> 'id_str':string?,
> 'name':string?,
> 'id':int64?
> }
> create type entitiesType1 as open {
> 'media':[listType3]?,
>
