jnturton commented on code in PR #2633:
URL: https://github.com/apache/drill/pull/2633#discussion_r957489232
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/parser/SimpleMessageParser.java:
##########
@@ -129,6 +135,44 @@ private boolean parseInnerLevel(TokenIterator tokenizer,
int level) throws Messa
return parseToElement(tokenizer, level + 1);
}
+ /**
+ * This function is called when a storage plugin needs to retrieve values
which have been read. This logic
+ * enables use of the data path in these situations. Normally, when the
datapath is defined, the JSON reader
+ * will "free-wheel" over unprojected columns or columns outside of the
datapath. However, in this case, often
+ * the values which are being read, are outside the dataPath. This logic
offers a way to capture these values
+ * without creating a ValueVector for them.
+ *
+ * @param tokenizer A {@link TokenIterator} of the parsed JSON data.
+ * @param fieldName A {@link String} of the pagination field name.
Review Comment:
```suggestion
* @param fieldName A {@link String} of the listener column name.
```
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/loader/TupleParser.java:
##########
@@ -127,10 +127,19 @@ public TupleParser(JsonLoaderImpl loader, TupleWriter
tupleWriter, TupleMetadata
@Override
public ElementParser onField(String key, TokenIterator tokenizer) {
- if (!tupleWriter.isProjected(key)) {
+ if (projectField(key)) {
+ return fieldParserFor(key, tokenizer);
+ } else {
return fieldFactory().ignoredFieldParser();
+ }
+ }
+
+ private boolean projectField(String key) {
+ // This method makes sure that fields necessary for pagination are read.
Review Comment:
```suggestion
// This method makes sure that fields necessary for column listeners are
read.
```
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/values/ScalarListener.java:
##########
@@ -76,4 +79,30 @@ protected void setArrayNull() {
protected UserException typeConversionError(String jsonType) {
return loader.typeConversionError(schema(), jsonType);
}
+
+ /**
+ * Adds a field's most recent value to the column listener map.
+ * This data is only stored if the listener column map is defined, and has
keys.
+ * @param key The key of the pagination field
Review Comment:
```suggestion
* @param key The key of the listener field
```
##########
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/values/ScalarListener.java:
##########
@@ -76,4 +79,30 @@ protected void setArrayNull() {
protected UserException typeConversionError(String jsonType) {
return loader.typeConversionError(schema(), jsonType);
}
+
+ /**
+ * Adds a field's most recent value to the column listener map.
+ * This data is only stored if the listener column map is defined, and has
keys.
+ * @param key The key of the pagination field
+ * @param value The value of to be retained
+ */
+ protected void addValueToListenerMap(String key, String value) {
+ Map<String,Object> listenerColumnMap = loader.listenerColumnMap();
+
+ if (listenerColumnMap == null || listenerColumnMap.isEmpty()) {
+ return;
+ } else if (listenerColumnMap.containsKey(key) &&
StringUtils.isNotEmpty(value)) {
+ listenerColumnMap.put(key, value);
+ }
+ }
+
+ protected void addValueToListenerMap(String key, Object value) {
+ Map<String, Object> paginationMap = loader.listenerColumnMap();
Review Comment:
```suggestion
Map<String, Object> listenerMap = loader.listenerColumnMap();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
