[
https://issues.apache.org/jira/browse/DRILL-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597256#comment-17597256
]
ASF GitHub Bot commented on DRILL-8287:
---------------------------------------
jnturton commented on code in PR #2633:
URL: https://github.com/apache/drill/pull/2633#discussion_r957489232
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/parser/SimpleMessageParser.java:
##########
@@ -129,6 +135,44 @@ private boolean parseInnerLevel(TokenIterator tokenizer,
int level) throws Messa
return parseToElement(tokenizer, level + 1);
}
+ /**
+ * This function is called when a storage plugin needs to retrieve values
which have been read. This logic
+ * enables use of the data path in these situations. Normally, when the
datapath is defined, the JSON reader
+ * will "free-wheel" over unprojected columns or columns outside of the
datapath. However, in this case, often
+ * the values which are being read, are outside the dataPath. This logic
offers a way to capture these values
+ * without creating a ValueVector for them.
+ *
+ * @param tokenizer A {@link TokenIterator} of the parsed JSON data.
+ * @param fieldName A {@link String} of the pagination field name.
Review Comment:
```suggestion
* @param fieldName A {@link String} of the listener column name.
```
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/TupleParser.java:
##########
@@ -127,10 +127,19 @@ public TupleParser(JsonLoaderImpl loader, TupleWriter
tupleWriter, TupleMetadata
@Override
public ElementParser onField(String key, TokenIterator tokenizer) {
- if (!tupleWriter.isProjected(key)) {
+ if (projectField(key)) {
+ return fieldParserFor(key, tokenizer);
+ } else {
return fieldFactory().ignoredFieldParser();
+ }
+ }
+
+ private boolean projectField(String key) {
+ // This method makes sure that fields necessary for pagination are read.
Review Comment:
```suggestion
// This method makes sure that fields necessary for column listeners are
read.
```
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/values/ScalarListener.java:
##########
@@ -76,4 +79,30 @@ protected void setArrayNull() {
protected UserException typeConversionError(String jsonType) {
return loader.typeConversionError(schema(), jsonType);
}
+
+ /**
+ * Adds a field's most recent value to the column listener map.
+ * This data is only stored if the listener column map is defined, and has
keys.
+ * @param key The key of the pagination field
Review Comment:
```suggestion
* @param key The key of the listener field
```
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/values/ScalarListener.java:
##########
@@ -76,4 +79,30 @@ protected void setArrayNull() {
protected UserException typeConversionError(String jsonType) {
return loader.typeConversionError(schema(), jsonType);
}
+
+ /**
+ * Adds a field's most recent value to the column listener map.
+ * This data is only stored if the listener column map is defined, and has
keys.
+ * @param key The key of the pagination field
+ * @param value The value of to be retained
+ */
+ protected void addValueToListenerMap(String key, String value) {
+ Map<String,Object> listenerColumnMap = loader.listenerColumnMap();
+
+ if (listenerColumnMap == null || listenerColumnMap.isEmpty()) {
+ return;
+ } else if (listenerColumnMap.containsKey(key) &&
StringUtils.isNotEmpty(value)) {
+ listenerColumnMap.put(key, value);
+ }
+ }
+
+ protected void addValueToListenerMap(String key, Object value) {
+ Map<String, Object> paginationMap = loader.listenerColumnMap();
Review Comment:
```suggestion
Map<String, Object> listenerMap = loader.listenerColumnMap();
```
> Add Support for Keyset Based Pagination
> ---------------------------------------
>
> Key: DRILL-8287
> URL: https://issues.apache.org/jira/browse/DRILL-8287
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - HTTP
> Affects Versions: 1.20.2
> Reporter: Charles Givre
> Assignee: Charles Givre
> Priority: Major
> Fix For: 2.0.0
>
>
> Some APIs such as HubSpot use values in the result set to indicate whether
> there are additional pages. This PR adds support for this kind of
> pagination. Note that current implementation only works for JSON based APIs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
