[ https://issues.apache.org/jira/browse/SPARK-36986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Boavida updated SPARK-36986: ------------------------------------ Description: Our spark usage, requires us to build an external schema and pass it on while creating a DataSet. While working through this, I found an optimization would improve greatly Spark's flexibility to query external schema management. Scope: ability to retrieve a field's name and schema in one single call, requesting to return a tupple by index. Means extending the StructType class to support an additional method This is what the function would look like: /** * Returns the index and field structure by name. * If it doesn't find it, returns None. * Avoids two client calls/loops to obtain consolidated field info. * */ def getIndexAndFieldByName(name: String): Option[(Int, StructField)] = \{ val field = nameToField.get(name) if(field.isDefined) { Some((fieldIndex(name), field.get)) } else \{ None } } This is particularly useful from an efficiency perspective, when we're parsing a Json structure and we want to check for every field what is the name and field type already defined in the schema I will create a corresponding branch for PR review, assuming that there are no concerns with the above proposal. was: Our spark usage, requires us to build an external schema and pass it on while creating a DataSet. While working through this, I found a couple of optimizations would improve greatly Spark's flexibility to handle external schema management. Scope: ability to retrieve a field's name and schema in one single call, requesting to return a tupple by index. Means extending the StructType class to support an additional method This is what the function would look like: /** * Returns the index and field structure by name. * If it doesn't find it, returns None. * Avoids two client calls/loops to obtain consolidated field info. * */ def getIndexAndFieldByName(name: String): Option[(Int, StructField)] = \{ val field = nameToField.get(name) if(field.isDefined) \{ Some((fieldIndex(name), field.get)) } else { None } } This is particularly useful from an efficiency perspective, when we're parsing a Json structure and we want to check for every field what is the name and field type already defined in the schema I will create a corresponding branch for PR review, assuming that there are no concerns with the above proposal. > Improving schema filtering flexibility > -------------------------------------- > > Key: SPARK-36986 > URL: https://issues.apache.org/jira/browse/SPARK-36986 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.3.0 > Reporter: Rodrigo Boavida > Priority: Major > > Our spark usage, requires us to build an external schema and pass it on while > creating a DataSet. > While working through this, I found an optimization would improve greatly > Spark's flexibility to query external schema management. > Scope: ability to retrieve a field's name and schema in one single call, > requesting to return a tupple by index. > Means extending the StructType class to support an additional method > This is what the function would look like: > /** > * Returns the index and field structure by name. > * If it doesn't find it, returns None. > * Avoids two client calls/loops to obtain consolidated field info. > * > */ > def getIndexAndFieldByName(name: String): Option[(Int, StructField)] = \{ > val field = nameToField.get(name) if(field.isDefined) { > Some((fieldIndex(name), field.get)) } > else \{ None } > } > This is particularly useful from an efficiency perspective, when we're > parsing a Json structure and we want to check for every field what is the > name and field type already defined in the schema > I will create a corresponding branch for PR review, assuming that there are > no concerns with the above proposal. > -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org