Change in asterixdb[master]: Add record reader and parser for CAP messages
Till Westmann has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 10: Also added Ian as a reviewer as he's currently working on https://issues.apache.org/jira/browse/ASTERIXDB-1714 . -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 10 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Add record reader and parser for CAP messages
Till Westmann has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 10: Indeed the license has been moved to category X [1] very recently and so we should replace the use of org.json code with Jackson which we have already started [2]. [1] https://www.apache.org/legal/resolved#category-x [2] https://github.com/apache/asterixdb/commit/4ad756062d7afde7c03801705dad4ca9854ccf87 -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 10 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Add record reader and parser for CAP messages
abdullah alamoudi has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 10: this change looks good to me. There is however one issue: http://www.json.org/license.html if I understand correctly, ASF will not accept releases which contain this license. unfortunately, I think we will have to find another library which provide the needed functionality. -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 10 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Fixed issues with Replication tests
Akshay Manchale has posted comments on this change. Change subject: Fixed issues with Replication tests .. Patch Set 3: (2 comments) https://asterix-gerrit.ics.uci.edu/#/c/1334/3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java File hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleReference.java: Line 92: return (int) Math.ceil(fieldCount / 4.0); > It is because records have 2 bits for each field to represent whether the f Upon further discussion with Ian, I found that the missing/null bytes and the values here are not related. I'm going to change it back to 8. https://asterix-gerrit.ics.uci.edu/#/c/1334/3/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java File hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/tuples/SimpleTupleWriter.java: Line 77: targetBuf[runner++] = (byte) 0xAA; > This initialization follows from how RecordBuilder is building records. See Changing this back to 0 -- To view, visit https://asterix-gerrit.ics.uci.edu/1334 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I204a41311012dc348d4d41ae8bf6ec8554ab3dc3 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Akshay Manchale Gerrit-Reviewer: Akshay Manchale Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-HasComments: Yes
Change in asterixdb[master]: Add record reader and parser for CAP messages
Jenkins has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 10: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1132/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 10 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Add record reader and parser for CAP messages
Jenkins has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 10: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1132/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 10 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Feed Connection Refactoring
abdullah alamoudi has posted comments on this change.
Change subject: Feed Connection Refactoring
..
Patch Set 5:
(17 comments)
Looks good overall and I can clearly see that you understand what is happening.
Please go over the comments. I will look at the new patch.
Cheers,
Abdullah.
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActiveManagerMessage.java
File
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActiveManagerMessage.java:
Line 31:
revert?
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
File
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java:
Line 116: private static final ILangCompilationProvider compilationProvider
= new AqlCompilationProvider();
get rid of static compilationProvider
and qtFactory and instead pass them as parameters!
Line 197: ingestionOp = new FeedIntakeOperatorDescriptor(jobSpec,
feed, firstOp.getAdaptorLibraryName(),
when will the adapterFactory be null?
Line 232: if (opDesc instanceof
AsterixLSMTreeInsertDeleteOperatorDescriptor
I believe that we don't need the if or the else here!!
ideally, we should get rid of the feed collect (you changed it to take input)
but it doesn't do anything essentially. we don't need to do this now. we can
create an issue to get rid of it.
Line 370: JobSpecification feedJob;
remove the feedJob variable?
move the declaration of ingestionLocations down to the assignment statement?
Line 386: feedJob = combineIntakeCollectJobs(metadataProvider, feed,
intakeJob, jobsList, feedConnections,
return immediately rather than assign then return
Line 391: private static Pair buildSendFeedMessageRuntime(
remove unused
Line 400:
remove unused... and remove the feed message operator completely
Line 405: }
remove unused
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
File
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java:
Line 345: case Statement.Kind.CONNECT_FEED:
add test cases that queries the connection dataset after connect and after
disconnect?
Line 2143: try {
what if feed has already been started?
fix and add a test case please :)
Line 2183: // TODO: check feed is actually running.
do the todo and add a test case?
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/FeedConnectionRequest.java
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/FeedConnectionRequest.java:
Line 95:
get rid of connection status as it is not needed anymore?
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/watch/FeedConnectJobInfo.java
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/watch/FeedConnectJobInfo.java:
Line 44: public FeedConnectJobInfo(EntityId entityId, JobId jobId,
ActivityState state, JobSpecification spec) {
rename this class?
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/FeedMessageOperatorDescriptor.java
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/operators/FeedMessageOperatorDescriptor.java:
this class should go away
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-lang-aql/src/main/javacc/AQL.jj
File asterixdb/asterix-lang-aql/src/main/javacc/AQL.jj:
do the same changes in the sqlpp.jj file?
https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
File asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj:
Line 1150: }
stop is missing?
--
To view, visit https://asterix-gerrit.ics.uci.edu/1259
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: abdullah alamoudi
Gerrit-HasComments: Yes
Change in asterixdb[master]: Add record reader and parser for CAP messages
Jenkins has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 10: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3298/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 10 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Add record reader and parser for CAP messages
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/1269 to look at the new patch set (#10). Change subject: Add record reader and parser for CAP messages .. Add record reader and parser for CAP messages 1. Add record reader for CAP message. 2. Add CAP message parser based on JSON library and ADMParser. 3. Fix ASTERIXDB-1690: deadlock between close() and take() in FileSystemWatcher 4. Add test cases for using CAP adaptor in feed and load statement. 5. Add an option in configuration to allow user use CAP adaptor loading certain XML documents. - "collection" = "true" (Multiple records in one document, check "document_with_records_collection.xml" as an example). Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da --- A asterixdb/asterix-app/data/cap_messages/ER.xml A asterixdb/asterix-app/data/cap_messages/HSA.xml A asterixdb/asterix-app/data/cap_messages/STA.xml A asterixdb/asterix-app/data/cap_messages/bigger.xml A asterixdb/asterix-app/data/cap_messages/small_ER.xml A asterixdb/asterix-app/data/document_with_records_collection.xml A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.5.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java 36 files changed, 1,255 insertions(+), 34 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/69/1269/10 -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.e
[jira] [Updated] (ASTERIXDB-1690) FileSystemWatcher.close() is blocked by FileSystemWatcher.take() when monitoring directory.
[
https://issues.apache.org/jira/browse/ASTERIXDB-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xikui Wang updated ASTERIXDB-1690:
--
Description:
When create feed for directory monitoring, disconnect feed can be blocked by
the FileSystemWatcher. When FileSystemWatcher is waiting for new files, the
whole object is locked by current thread. So when another thread trying to
close the watcher, it will be blocked. Here is the take() method from
FileSystemWatcher:
{noformat}
public synchronized File take() throws IOException {
File next = poll();
if (next != null) {
return next;
}
if (done || !isFeed) {
return null;
}
// No file was found, wait for the filesystem to push events
WatchKey key = null;
lock.lock();
try {
while (!it.hasNext()) {
try {
key = watcher.take();
} catch (InterruptedException x) {
if (LOGGER.isEnabledFor(Level.WARN)) {
LOGGER.warn("Feed Closed");
}
if (watcher == null) {
return null;
}
continue;
} catch (ClosedWatchServiceException e) {
if (LOGGER.isEnabledFor(Level.WARN)) {
LOGGER.warn("The watcher has exited");
}
if (watcher == null) {
return null;
}
continue;
}
handleEvents(key);
if (endOfEvents(key)) {
return null;
}
}
} finally {
lock.unlock();
}
// files were found, re-create the iterator and move it one step
return it.next();
}
{noformat}
was:
When create feed for directory monitoring, disconnect feed can be blocked by
the FileSystemWatcher. When FileSystemWatcher is waiting for new files, the
whole object is locked by current thread. So when another thread trying to
close the watcher, it will be blocked. Here is the take() method from
FileSystemWatcher:
public synchronized File take() throws IOException {
File next = poll();
if (next != null) {
return next;
}
if (done || !isFeed) {
return null;
}
// No file was found, wait for the filesystem to push events
WatchKey key = null;
lock.lock();
try {
while (!it.hasNext()) {
try {
key = watcher.take();
} catch (InterruptedException x) {
if (LOGGER.isEnabledFor(Level.WARN)) {
LOGGER.warn("Feed Closed");
}
if (watcher == null) {
return null;
}
continue;
} catch (ClosedWatchServiceException e) {
if (LOGGER.isEnabledFor(Level.WARN)) {
LOGGER.warn("The watcher has exited");
}
if (watcher == null) {
return null;
}
continue;
}
handleEvents(key);
if (endOfEvents(key)) {
return null;
}
}
} finally {
lock.unlock();
}
// files were found, re-create the iterator and move it one step
return it.next();
}
> FileSystemWatcher.close() is blocked by FileSystemWatcher.take() when
> monitoring directory.
> ---
>
> Key: ASTERIXDB-1690
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1690
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Feeds
>Reporter: Xikui Wang
>Assignee: Xikui Wang
>
> When create feed for directory monitoring, disconnect feed can be blocked by
> the FileSystemWatcher. When FileSystemWatcher is waiting for new files, the
> whole object is locked by current thread. So when another thread trying to
> close the watcher, it will be blocked. Here is the take() method from
> FileSystemWatcher:
> {noformat}
> public synchronized File take() throws IOException {
> File next = poll();
> if (next != null) {
> return next;
> }
> if (done || !isFeed) {
> return null;
> }
> // No file was found, wait for the filesystem to push events
> WatchKey key = null;
> lock.lock();
> try {
> while (!it.hasNext()) {
> try {
> ke
Change in asterixdb[master]: Add record reader and parser for CAP messages
Xikui Wang has posted comments on this change. Change subject: Add record reader and parser for CAP messages .. Patch Set 9: @Till & Abdullah, One thing I am not very sure in this patch is ASTERIXDB-1690. If you can help me verify that, it will be very helpful. :) -- To view, visit https://asterix-gerrit.ics.uci.edu/1269 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da Gerrit-PatchSet: 9 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Xikui Wang Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
[jira] [Resolved] (ASTERIXDB-1609) TwitterParser does not parse closed-nullable fields.
[
https://issues.apache.org/jira/browse/ASTERIXDB-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xikui Wang resolved ASTERIXDB-1609.
---
Resolution: Fixed
> TwitterParser does not parse closed-nullable fields.
>
>
> Key: ASTERIXDB-1609
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1609
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Data Model, Feeds
>Reporter: Wail Alkowaileet
>Assignee: Xikui Wang
>
> TwitterParser doesn't support types NULL/MISSING
> To reproduce, connect TwitterFeed using the defined type below. I
> roundtrip-ed it using load statement and it was successfully loaded.
> DDL:
> {noformat}
> drop dataverse feeds if exists;
> create dataverse feeds;
> use dataverse feeds;
> create type userType1 as open {
> 'utc_offset':int64?,
> 'friends_count':int64?,
> 'profile_image_url_https':string?,
> 'listed_count':int64?,
> 'profile_background_image_url':string?,
> 'default_profile_image':boolean?,
> 'favourites_count':int64?,
> 'description':string?,
> 'created_at':string?,
> 'is_translator':boolean?,
> 'profile_background_image_url_https':string?,
> 'protected':boolean?,
> 'screen_name':string?,
> 'id_str':string?,
> 'profile_link_color':string?,
> 'id':int64?,
> 'geo_enabled':boolean?,
> 'profile_background_color':string?,
> 'lang':string?,
> 'profile_sidebar_border_color':string?,
> 'profile_text_color':string?,
> 'verified':boolean?,
> 'profile_image_url':string?,
> 'time_zone':string?,
> 'contributors_enabled':boolean?,
> 'profile_background_tile':boolean?,
> 'profile_banner_url':string?,
> 'statuses_count':int64?,
> 'followers_count':int64?,
> 'profile_use_background_image':boolean?,
> 'default_profile':boolean?,
> 'name':string?,
> 'location':string?,
> 'profile_sidebar_fill_color':string?,
> 'url':string?
> }
> create type smallType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type largeType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type thumbType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type mediumType1 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type sizesType1 as open {
> 'small':smallType1?,
> 'large':largeType1?,
> 'thumb':thumbType1?,
> 'medium':mediumType1?
> }
> create type listType2 as open {
> 'content_type':string?,
> 'bitrate':int64?,
> 'url':string?
> }
> create type video_infoType1 as open {
> 'aspect_ratio':[int64]?,
> 'duration_millis':int64?,
> 'variants':[listType2]?
> }
> create type listType1 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'sizes':sizesType1?,
> 'id_str':string?,
> 'expanded_url':string?,
> 'media_url_https':string?,
> 'id':int64?,
> 'type':string?,
> 'media_url':string?,
> 'url':string?,
> 'video_info':video_infoType1?,
> 'source_user_id':int64?,
> 'source_status_id':int64?,
> 'source_status_id_str':string?,
> 'source_user_id_str':string?
> }
> create type extended_entitiesType1 as open {
> 'media':[listType1]?
> }
> create type smallType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type largeType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type thumbType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type mediumType2 as open {
> 'w':int64?,
> 'h':int64?,
> 'resize':string?
> }
> create type sizesType2 as open {
> 'small':smallType2?,
> 'large':largeType2?,
> 'thumb':thumbType2?,
> 'medium':mediumType2?
> }
> create type listType3 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'sizes':sizesType2?,
> 'id_str':string?,
> 'expanded_url':string?,
> 'media_url_https':string?,
> 'id':int64?,
> 'type':string?,
> 'media_url':string?,
> 'url':string?,
> 'source_user_id':int64?,
> 'source_status_id':int64?,
> 'source_status_id_str':string?,
> 'source_user_id_str':string?
> }
> create type listType4 as open {
> 'display_url':string?,
> 'indices':[int64]?,
> 'expanded_url':string?,
> 'url':string?
> }
> create type listType5 as open {
> 'indices':[int64]?,
> 'text':string?
> }
> create type listType6 as open {
> 'indices':[int64]?,
> 'screen_name':string?,
> 'id_str':string?,
> 'name':string?,
> 'id':int64?
> }
> create type entitiesType1 as open {
> 'media':[listType3]?,
> 'urls':[listType4]?,
> 'hashtags':[listType5]?,
> 'user_mentions':[listType6]?
> }
[jira] [Resolved] (ASTERIXDB-1709) FileFeedSocketAdapterClient connect socket feed without check availability
[ https://issues.apache.org/jira/browse/ASTERIXDB-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xikui Wang resolved ASTERIXDB-1709. --- Resolution: Fixed > FileFeedSocketAdapterClient connect socket feed without check availability > -- > > Key: ASTERIXDB-1709 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1709 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Xikui Wang >Assignee: Xikui Wang >Priority: Trivial > > When a FileFeedSocketAdapterClient starts, it will try to connect socket feed > intermediately. It will cause sporadic connection error, if the socket server > is not started yet. > Delay the start of this connection can avoid this problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (ASTERIXDB-1690) FileSystemWatcher.close() is blocked by FileSystemWatcher.take() when monitoring directory.
[
https://issues.apache.org/jira/browse/ASTERIXDB-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xikui Wang reassigned ASTERIXDB-1690:
-
Assignee: Xikui Wang
> FileSystemWatcher.close() is blocked by FileSystemWatcher.take() when
> monitoring directory.
> ---
>
> Key: ASTERIXDB-1690
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1690
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Feeds
>Reporter: Xikui Wang
>Assignee: Xikui Wang
>
> When create feed for directory monitoring, disconnect feed can be blocked by
> the FileSystemWatcher. When FileSystemWatcher is waiting for new files, the
> whole object is locked by current thread. So when another thread trying to
> close the watcher, it will be blocked. Here is the take() method from
> FileSystemWatcher:
> public synchronized File take() throws IOException {
> File next = poll();
> if (next != null) {
> return next;
> }
> if (done || !isFeed) {
> return null;
> }
> // No file was found, wait for the filesystem to push events
> WatchKey key = null;
> lock.lock();
> try {
> while (!it.hasNext()) {
> try {
> key = watcher.take();
> } catch (InterruptedException x) {
> if (LOGGER.isEnabledFor(Level.WARN)) {
> LOGGER.warn("Feed Closed");
> }
> if (watcher == null) {
> return null;
> }
> continue;
> } catch (ClosedWatchServiceException e) {
> if (LOGGER.isEnabledFor(Level.WARN)) {
> LOGGER.warn("The watcher has exited");
> }
> if (watcher == null) {
> return null;
> }
> continue;
> }
> handleEvents(key);
> if (endOfEvents(key)) {
> return null;
> }
> }
> } finally {
> lock.unlock();
> }
> // files were found, re-create the iterator and move it one step
> return it.next();
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Change in asterixdb[master]: Enable extension of IDataFormat
Jenkins has posted comments on this change. Change subject: Enable extension of IDataFormat .. Patch Set 6: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1131/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
Steven Jacobs has posted comments on this change. Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser .. Patch Set 7: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/1339 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Steven Jacobs Gerrit-Reviewer: Wail Alkowaileet Gerrit-Reviewer: Xikui Wang Gerrit-HasComments: No
Change in asterixdb[master]: Enable extension of IDataFormat
Jenkins has posted comments on this change. Change subject: Enable extension of IDataFormat .. Patch Set 6: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1131/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Enable extension of IDataFormat
Jenkins has posted comments on this change. Change subject: Enable extension of IDataFormat .. Patch Set 6: Integration Tests Aborted https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1130/ : ABORTED -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Enable extension of IDataFormat
Jenkins has posted comments on this change. Change subject: Enable extension of IDataFormat .. Patch Set 6: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1130/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Enable extension of IDataFormat
Jenkins has posted comments on this change. Change subject: Enable extension of IDataFormat .. Patch Set 6: WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN: * asterixdb * hyracks-fullstack PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES! -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Enable extension of IDataFormat
Jenkins has posted comments on this change. Change subject: Enable extension of IDataFormat .. Patch Set 6: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3297/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: Enable extension of IDataFormat
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/1349 to look at the new patch set (#6). Change subject: Enable extension of IDataFormat .. Enable extension of IDataFormat Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/ILangCompilationProvider.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/ConstantFoldingRule.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/APIFramework.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ConnectorAPIServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/cc/CompilerExtensionManager.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/cc/IStatementExecutorExtension.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/DefaultStatementExecutorFactory.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplicationEntryPoint.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/GlobalRecoveryManager.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/ConnectorAPIServletTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/aql/translator/QueryTranslatorTest.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/IAsterixApplicationContextInfo.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/AqlMetadataProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtils.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/formats/base/IDataFormat.java A asterixdb/asterix-om/src/main/java/org/apache/asterix/formats/base/IDataFormatProvider.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/formats/NonTaggedDataFormat.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/util/AsterixAppContextInfo.java M hyracks-fullstack/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/api/AbstractCompilerFactoryBuilder.java M hyracks-fullstack/algebricks/algebricks-compiler/src/main/java/org/apache/hyracks/algebricks/compiler/api/HeuristicCompilerFactoryBuilder.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/HashPartitionExchangePOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/HashPartitionMergeExchangePOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/SortMergeExchangePOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/jobgen/impl/JobGenContext.java 25 files changed, 208 insertions(+), 128 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/49/1349/6 -- To view, visit https://asterix-gerrit.ics.uci.edu/1349 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b3e26fbb4a5da9ae4522b228991155f1a06d854 Gerrit-PatchSet: 6 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: abdullah alamoudi Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-Reviewer: abdullah alamoudi
Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
Wail Alkowaileet has posted comments on this change. Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser .. Patch Set 7: Code-Review+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/1339 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Wail Alkowaileet Gerrit-Reviewer: Xikui Wang Gerrit-HasComments: No
Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
Jenkins has posted comments on this change. Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser .. Patch Set 7: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1129/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1339 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd Gerrit-PatchSet: 7 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Wail Alkowaileet Gerrit-Reviewer: Xikui Wang Gerrit-HasComments: No
