[GitHub] spark issue #18376: [SPARK-21161][CORE] Make the parseHostPort method tolera...
Github user janplus commented on the issue: https://github.com/apache/spark/pull/18376 Hi @srowen, @jiangxb1987. Thanks for quick response. After further debug and consideration, this fix should be done in the `spark-solr` project. Since this is a Solr specific issue. I'll close the PR and related issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18376: [SPARK-21161][CORE] Make the parseHostPort method...
Github user janplus closed the pull request at: https://github.com/apache/spark/pull/18376 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18376: [SPARK-21161][CORE] Make the parseHostPort method...
GitHub user janplus opened a pull request: https://github.com/apache/spark/pull/18376 [SPARK-21161][CORE] Make the parseHostPort method tolerate non digital charactors ## What changes were proposed in this pull request? Fix the following port parse bug when querying Solr data. ``` 17/06/21 12:40:53 INFO ContextLauncher: 17/06/21 12:40:53 ERROR scheduler.DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting down SparkContext 17/06/21 12:40:53 INFO ContextLauncher: java.lang.NumberFormatException: For input string: â8983_solrâ 17/06/21 12:40:53 INFO ContextLauncher: at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) 17/06/21 12:40:53 INFO ContextLauncher: at java.lang.Integer.parseInt(Integer.java:580) 17/06/21 12:40:53 INFO ContextLauncher: at java.lang.Integer.parseInt(Integer.java:615) 17/06/21 12:40:53 INFO ContextLauncher: at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272) 17/06/21 12:40:53 INFO ContextLauncher: at scala.collection.immutable.StringOps.toInt(StringOps.scala:29) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.util.Utils$.parseHostPort(Utils.scala:959) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.cluster.YarnScheduler.getRackForHost(YarnScheduler.scala:36) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:200) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSetManager$$anonfun$org$apache$spark$scheduler$TaskSetManager$$addPendingTask$1.apply(TaskSetManager.scala:181) 17/06/21 12:40:53 INFO ContextLauncher: at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 17/06/21 12:40:53 INFO ContextLauncher: at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSetManager.org$apache$spark$scheduler$TaskSetManager$$addPendingTask(TaskSetManager.scala:181) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSetManager$$anonfun$1.apply$mcVI$sp(TaskSetManager.scala:160) 17/06/21 12:40:53 INFO ContextLauncher: at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSetManager.(TaskSetManager.scala:159) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSchedulerImpl.createTaskSetManager(TaskSchedulerImpl.scala:212) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.TaskSchedulerImpl.submitTasks(TaskSchedulerImpl.scala:176) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1043) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:918) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:921) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:920) 17/06/21 12:40:53 INFO ContextLauncher: at scala.collection.immutable.List.foreach(List.scala:381) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:920) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:862) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1613) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594) 17/06/21 12:40:53 INFO ContextLauncher: at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) ``` ## How was this patch tested? Unit tests added. Also did integrated test with Solr. You can merge this pull request into a Git repository by running: $ git pull https://github.com/janplus/spark solr-port-bug-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18376.patch To close this pull request, make a commit to your master/trunk branch with (at least
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 Thanks @rxin @dongjoon-hyun @cloud-fan @liancheng I've learnt a lot from this PR! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 It seems failed the `org.apache.spark.sql.sources.CreateTableAsSelectSuite.create a table, drop it and create another one with the same name` test which is irrelevant with this PR. Maybe we should retest this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 cc @cloud-fan Thank you. I have resolved conflicts with master and done some code style fixes as you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70037559
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,154 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ExpectsInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => if (url ne null) getUrl(url) else
null
--- End diff --
Oh yes, it's simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70033835
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
--- End diff --
Well, I have missed this comment and finished the change...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 cc @rxin @cloud-fan Thank you for review I add a new commit doing the following things: 1. Use ExpectsInputTypes instead of ImplicitCastInputTypes. 2. Add some cases for invalid-type parameters. 3. Code style fixes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70028685
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
--- End diff --
OK, I'll use ExpectsInputTypes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70028094
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => if (url ne null) getUrl(url) else
null
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => if (key ne null) getPattern(key)
else null
+case _ => null
+ }
+
+ // If the partToExtract is a constant, cache the Extract part function
so that we don't need
+ // to check the partToExtract for every row.
+ @transient private lazy val cachedExtractPartFunc = children(1) match {
+case Literal(part: UTF8String, _) => getExtractPartFunc(part)
+case _ => null
+ }
+
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def getExtractPartFunc(partToExtract: UTF8String): (URL) =>
String = {
+partToExtract match {
+ case HOST => (url: URL) => url.getHost
+ case PATH => (url: URL) => url.getPath
+ case QUERY => (url: URL) => url.getQuery
+ case REF => (url: URL) => url.getRef
+ case PROTOCOL => (url: URL) => url.getProtocol
+ case FILE => (url: URL) => url.getFile
+ case AUTHORITY => (url: URL) => url.getAuthority
+ case USERINFO => (url: URL) => url.getUserInfo
+ case _ => (url: URL) => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8Stri
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70028081
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
--- End diff --
I am trying to make spark's behavior mostly like hive.
As hive does implicit cast for `key`, eg
> hive> select parse_url("http://spark/path?1=v;, "QUERY", 1);
v
Should we keep the same in spark?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70027444
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
--- End diff --
Yes, # is not a valid URL key. And I agree with you on not using a hacky
value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70026365
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => if (url ne null) getUrl(url) else
null
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => if (key ne null) getPattern(key)
else null
+case _ => null
+ }
+
+ // If the partToExtract is a constant, cache the Extract part function
so that we don't need
+ // to check the partToExtract for every row.
+ @transient private lazy val cachedExtractPartFunc = children(1) match {
+case Literal(part: UTF8String, _) => getExtractPartFunc(part)
+case _ => null
+ }
+
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def getExtractPartFunc(partToExtract: UTF8String): (URL) =>
String = {
+partToExtract match {
+ case HOST => (url: URL) => url.getHost
--- End diff --
Oh, yes, it is much more concise.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70026344
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => if (url ne null) getUrl(url) else
null
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => if (key ne null) getPattern(key)
else null
+case _ => null
+ }
+
+ // If the partToExtract is a constant, cache the Extract part function
so that we don't need
+ // to check the partToExtract for every row.
+ @transient private lazy val cachedExtractPartFunc = children(1) match {
+case Literal(part: UTF8String, _) => getExtractPartFunc(part)
+case _ => null
+ }
+
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def getExtractPartFunc(partToExtract: UTF8String): (URL) =>
String = {
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70021325
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,152 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
--- End diff --
As I explained before, I can hardly find a magic `key` that may let us
treat `parse_url(url, part, magic key)` as `parse_url(url, part)`. I have doubt
on empty string, eg.
> hive> select parse_url("http://spark/path?=1;, "QUERY", "");
1
>hive> select parse_url("http://spark/path?=1;, "QUERY");
=1
Any suggestion on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 cc @cloud-fan @rxin @liancheng I did optimization for Literal `part`, so we don't need to check for every row. But since we may not assume in all circumstances the `part` is Literal, I keep the result being `null` when `part` is invalid. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r70018330
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,145 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
--- End diff --
Complication with the error
`Error:(686, 9) annotation argument needs to be a constant; found:
scala.this.Predef.augmentString`
So I should probably remain the current extended description string.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69933267
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,145 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
Good point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69932808
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,145 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
--- End diff --
OK, I'll fix this, thank you,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69932567
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8String):
UTF8String = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
--- End diff --
Since check it here is at Excutor side, it will be of no difference
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69905666
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,145 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
Test in 2.10.5 too. The result is the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69894466
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8String):
UTF8String = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
--- End diff --
I've though this again, row level check seems inevitable. Since we
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69857849
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8String):
UTF8String = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
--- End diff --
Before I do this change, I need to confirm the behavior once more
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69857153
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,145 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
OK
> scala> val key: UTF8String = null
key: org.apache.spark.unsafe.types.UTF8String = null
> scala> Literal.create(key, StringType) match {
| case Literal(k: UTF8String, _) => true
| case _ => false
| }
res13: Boolean = false
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69855494
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,145 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = children(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = children(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
I think `null` will not match this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69855302
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8String):
UTF8String = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
--- End diff --
Hmm, make sense, I'll investigate on this. Thank you.
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69851574
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,52 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal(urlStr), Literal(partToExtract))), expected)
+}
+def checkParseUrlWithKey(
+expected: String,
+urlStr: String,
+partToExtract: String,
+key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal(urlStr), Literal(partToExtract),
Literal(key))), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("userinfo",
"http://useri...@spark.apache.org/path?query=1;, "USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "USERINFO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "")
+
+// exceptional cases
+intercept[java.util.regex.PatternSyntaxException] {
+ evaluate(ParseUrl(Seq(Literal("http://spark.apache.org/path?;),
+Literal("QUERY"), Literal("???"
+}
+
+// arguments checking
+assert(ParseUrl(Seq(Literal("1"))).checkInputDataTypes().isFailure)
+assert(ParseUrl(Seq(Literal("1"), Literal("2"), Literal("3"),
Literal("4")))
--- End diff --
As I declare ParseUrl with `ImplicitCastInputTypes`, I am no sure whether
the cases with invalid-type parameters is necessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69851306
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8String):
UTF8String = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
+}
+ }
+
+ private def parseUrlWithoutKey(url: UTF8String, partToExtract:
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69851163
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
--- End diff --
OK..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69851065
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,160 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ Pattern.compile(REGEXPREFIX + key.toString + REGEXSUBFIX)
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+ private def extractValueFromQuery(query: UTF8String, pattern: Pattern):
UTF8String = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: UTF8String):
UTF8String = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
+}
+ }
+
+ private def parseUrlWithoutKey(url: UTF8String, partToExtract:
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 Resolve conflicts with master. cc @cloud-fan , is there any more question about this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69532511
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
+ Literal.create(partToExtract, StringType))), expected)
+}
+def checkParseUrlWithKey(
+expected: String, urlStr: String,
+partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType))), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("userinfo",
"http://useri...@spark.apache.org/path?query=1;, "USERINFO")
--- End diff --
Then the result is `null`. I'll add a test case for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69532282
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
+ Literal.create(partToExtract, StringType))), expected)
+}
+def checkParseUrlWithKey(
+expected: String, urlStr: String,
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69532226
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69477433
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ val sb = new StringBuilder()
+ sb.append(REGEXPREFIX).append(key.toString).append(REGEXSUBFIX)
+ Pattern.compile(sb.toString())
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+
+ private def extractValueFromQuery(query: Any, pattern: Pattern): Any = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: Any): Any = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
+}
+ }
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69476414
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ val sb = new StringBuilder()
+ sb.append(REGEXPREFIX).append(key.toString).append(REGEXSUBFIX)
+ Pattern.compile(sb.toString())
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: UTF8String): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case e: MalformedURLException => null
+}
+ }
+
+
+ private def extractValueFromQuery(query: Any, pattern: Pattern): Any = {
--- End diff --
Oh, get your point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69475224
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +654,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: UTF8String): Pattern = {
+if (key != null) {
+ val sb = new StringBuilder()
+ sb.append(REGEXPREFIX).append(key.toString).append(REGEXSUBFIX)
--- End diff --
Fine. I'll fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 @cloud-fan Thank you for review. I did some code style fixes as you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69408201
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: Any): Pattern = {
+if (key != null) {
+ val sb = new StringBuilder()
+ sb.append(REGEXPREFIX).append(key.toString).append(REGEXSUBFIX)
+ Pattern.compile(sb.toString())
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: Any): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case NonFatal(_) => null
+}
+ }
+
+
+ private def extractValueFromQuery(query: Any, pattern: Pattern): Any = {
+val m = pattern.matcher(query.toString)
+if (m.find()) {
+ UTF8String.fromString(m.group(2))
+} else {
+ null
+}
+ }
+
+ private def extractFromUrl(url: URL, partToExtract: Any): Any = {
+if (partToExtract.equals(HOST)) {
+ UTF8String.fromString(url.getHost)
+} else if (partToExtract.equals(PATH)) {
+ UTF8String.fromString(url.getPath)
+} else if (partToExtract.equals(QUERY)) {
+ UTF8String.fromString(url.getQuery)
+} else if (partToExtract.equals(REF)) {
+ UTF8String.fromString(url.getRef)
+} else if (partToExtract.equals(PROTOCOL)) {
+ UTF8String.fromString(url.getProtocol)
+} else if (partToExtract.equals(FILE)) {
+ UTF8String.fromString(url.getFile)
+} else if (partToExtract.equals(AUTHORITY)) {
+ UTF8String.fromString(url.getAuthority)
+} else if (partToExtract.equals(USERINFO)) {
+ UTF8String.fromString(url.getUserInfo)
+} else {
+ null
+}
+ }
+
+ private def parseUrlWi
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69408152
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: Any): Pattern = {
+if (key != null) {
+ val sb = new StringBuilder()
+ sb.append(REGEXPREFIX).append(key.toString).append(REGEXSUBFIX)
+ Pattern.compile(sb.toString())
+} else {
+ null
+}
+ }
+
+ private def getUrl(url: Any): URL = {
+try {
+ new URL(url.toString)
+} catch {
+ case NonFatal(_) => null
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69408137
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure(s"$prettyName function requires two
or three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ private def getPattern(key: Any): Pattern = {
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 @dongjoon-hyun @cloud-fan It is nice to have you review my PR. Thank you! I have add a new commit with following things: 1. Revert driver side's literal key invalidation. 2. Resolve conflicts with master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69401574
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
+ Literal.create(partToExtract, StringType))), expected)
+}
+def checkParseUrlWithKey(
+expected: String, urlStr: String,
+partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType))), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("userinfo",
"http://useri...@spark.apache.org/path?query=1;, "USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "")
+
+// exceptional cases
+intercept[java.util.regex.PatternSyntaxException] {
--- End diff --
OK, @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 @dongjoon-hyun Thank you for review I have add a new commit which does following things: 1. Implement **Literal** `key` validation in `Driver` side. 2. Correct some failure message. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69386958
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
Thanks, this is my approach. It will give following message
> org.apache.spark.sql.AnalysisException: cannot resolve
'parse_url("http://spark.apache.org/path?;, "QUERY", "???")' due to data type
mismatch: invalid key '???'; line 1 pos 7
Does it look ok to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69386564
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
I get your point. Since implement here will still be `Executor` side, I'll
try to find another way to do this.
Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69386305
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
+case _ => null
+ }
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("parse_url function requires two or
three arguments")
--- End diff --
OK, thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69386280
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
BTW, implement here seems that the invalidation still happened in
`Executor` side. IMO, the invalidation of key's value can hardly be considered
as `AnalysisException`, just like the `rlike` function, isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69386079
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,163 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')
+ 'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')
+ 'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')
+ '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+ override def prettyName: String = "parse_url"
+
+ // If the url is a constant, cache the URL object so that we don't need
to convert url
+ // from UTF8String to String to URL for every row.
+ @transient private lazy val cachedUrl = stringExprs(0) match {
+case Literal(url: UTF8String, _) => getUrl(url)
+case _ => null
+ }
+
+ // If the key is a constant, cache the Pattern object so that we don't
need to convert key
+ // from UTF8String to String to StringBuilder to String to Pattern for
every row.
+ @transient private lazy val cachedPattern = stringExprs(2) match {
+case Literal(key: UTF8String, _) => getPattern(key)
--- End diff --
Oh, yes, I can do validation here.
But still we will have different exception types for **Literal** `key` and
**Non Literal** `key`.
That is `AnalysisException` for invalid Literal `key`, and
`PatternSyntaxException` for invalid non Literal `key`.
Will this be OK?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69385849
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
+ Literal.create(partToExtract, StringType))), expected)
+}
+def checkParseUrlWithKey(
+expected: String, urlStr: String,
+partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType))), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("userinfo",
"http://useri...@spark.apache.org/path?query=1;, "USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "")
+
+// exceptional cases
+intercept[java.util.regex.PatternSyntaxException] {
--- End diff --
Yes, definitely I can do that. In fact I have finished it.
But before I do the commit, let us get thought it first.
In `checkAnalysis` method for `LogicalPlan`, the only method will be called
for `Expression` is `checkInputDataTypes`
https://github.com/apache/spark/blob/d1e8108854deba3de8e2d87eb4389d11fb17ee57/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala#L64
Which means we can only implement this validation in `checkInputDataTypes`
of `ParseUrl`. In that circumstance spark will give the AnalysisException like
this
> org.apache.spark.sql.AnalysisException: cannot resolve
'parse_url("http://spark.apache.org/path?;, "QUERY", "???")' due to data type
mismatch: wrong key "???"; line 1 pos 0
But obviously this should not be a data type mismatch. This message may
confuse the users. Also the different message for **Literal** `key` and **Not
Literal** `key` may make them confused too.
Otherwise, if we do not validate the **Literal** `key`, the `Executor` will
get an exception at the first row. It seems not that unacceptable.
So compared the both sides, I think we should not do the Literal `key`
validation.
How do you think about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69385193
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
+ Literal.create(partToExtract, StringType))), expected)
+}
+def checkParseUrlWithKey(
+expected: String, urlStr: String,
+partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType))), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("userinfo",
"http://useri...@spark.apache.org/path?query=1;, "USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "")
+
+// exceptional cases
+intercept[java.util.regex.PatternSyntaxException] {
--- End diff --
Hi, @dongjoon-hyun
It seems only when `url`, `partToExtract` and `key` is all `Literal`, then
hive may give a `SemanticException`.
> hive> select * from url_parse_data;
OK
http://spark/path? QUERY ???
Time taken: 0.054 seconds, Fetched: 1 row(s)
> hive> select parse_url("http://spark/path?;, "QUERY", "???") from
url_parse_data;
FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '"???"':
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method
public java.lang.String
org.apache.hadoop.hive.ql.udf.UDFParseUrl.evaluate(java.lang.String,java.lang.String,java.lang.String)
on object org.apache.hadoop.hive.ql.udf.UDFParseUrl@59e082f8 of class
org.apache.hadoop.hive.ql.udf.UDFParseUrl with arguments
{http://spark/path?:java.lang.String, QUERY:java.lang.String,
???:java.lang.String} of size 3
> hive> select parse_url(url, "QUERY", "???") from url_parse_data;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
execute method public java.lang.String
org.apache.hadoop.hive.ql.udf.UDFParseUrl.evaluate(java.lang.String,java.lang.String,java.lang.String)
on object org.apache.hadoop.hive.ql.udf.UDFParseUrl@7d1f3fe9 of class
org.apache.hadoop.hive.ql.udf.UDFParseUrl with arguments
{http://spark/path?:java.lang.String, QUERY:java.lang.String,
???:java.lang.String} of size 3
> hive> select parse_url("http://spark/path?;, part, "???") from
url_parse_data;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69384848
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,51 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
+ Literal.create(partToExtract, StringType))), expected)
+}
+def checkParseUrlWithKey(
+expected: String, urlStr: String,
+partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Seq(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType))), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("userinfo",
"http://useri...@spark.apache.org/path?query=1;, "USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "")
+
+// exceptional cases
+intercept[java.util.regex.PatternSyntaxException] {
--- End diff --
I'll have a investigation on this.
It should be different whether `key` is `Literal`.
> hive> select parse_url("http://spark/path?;, "QUERY", "???");
FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '"???"':
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method
public java.lang.String
org.apache.hadoop.hive.ql.udf.UDFParseUrl.evaluate(java.lang.String,java.lang.String,java.lang.String)
on object org.apache.hadoop.hive.ql.udf.UDFParseUrl@6682e6a5 of class
org.apache.hadoop.hive.ql.udf.UDFParseUrl with arguments
{http://spark/path?:java.lang.String, QUERY:java.lang.String,
???:java.lang.String} of size 3
>
> hive> select parse_url("http://spark/path?;, "QUERY", name) from test;
OK
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
execute method public java.lang.String
org.apache.hadoop.hive.ql.udf.UDFParseUrl.evaluate(java.lang.String,java.lang.String,java.lang.String)
on object org.apache.hadoop.hive.ql.udf.UDFParseUrl@2035d65b of class
org.apache.hadoop.hive.ql.udf.UDFParseUrl with arguments
{http://spark/path?:java.lang.String, QUERY:java.lang.String,
???:java.lang.String} of size 3
Time taken: 0.039 seconds
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 @dongjoon-hyun and @cloud-fan Thank you for review. I have add a new commit which does following things: 1. Cache the url and the key pattern use the similar approach of `XPathBoolean` 2. Fix code style problems 3. Put the constants into an object 4. Add exceptional cases for invalid key --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69371935
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,129 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("parse_url function requires two or
three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ def parseUrlWithoutKey(url: Any, partToExtract: Any): Any = {
+if (url == null || partToExtract == null) {
+ null
+} else {
+ if (lastUrlStr == null || !url.equals(lastUrlStr)) {
+try {
+ lastUrl = new URL(url.toString)
+ lastUrlStr = url.asInstanceOf[UTF8String].clone()
+} catch {
+ case NonFatal(_) => return null
--- End diff --
Yes, it only throws `MalformedURLException`. But Dongjoon Hyun suggested me
to use `NonFatal(_)` here, as quoted
> We can use the following like Hive uses Exception. SecurityException
occurs possibly as a RuntimeException.
> `case NonFatal(_) =>`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69371498
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,129 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("parse_url function requires two or
three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ def parseUrlWithoutKey(url: Any, partToExtract: Any): Any = {
+if (url == null || partToExtract == null) {
+ null
+} else {
+ if (lastUrlStr == null || !url.equals(lastUrlStr)) {
+try {
+ lastUrl = new URL(url.toString)
+ lastUrlStr = url.asInstanceOf[UTF8String].clone()
--- End diff --
Em, good point. I'll try to find a better way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69371479
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -652,6 +656,129 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
override def prettyName: String = "rpad"
}
+object ParseUrl {
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+}
+
+/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+
+ private lazy val stringExprs = children.toArray
+ import ParseUrl._
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("parse_url function requires two or
three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ def parseUrlWithoutKey(url: Any, partToExtract: Any): Any = {
+if (url == null || partToExtract == null) {
+ null
+} else {
+ if (lastUrlStr == null || !url.equals(lastUrlStr)) {
+try {
+ lastUrl = new URL(url.toString)
+ lastUrlStr = url.asInstanceOf[UTF8String].clone()
+} catch {
+ case NonFatal(_) => return null
--- End diff --
As hive returns null for invalid urls, it seems reasonable that spark does
the same thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69370542
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,43 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType)),
+expected)
+}
+def checkParseUrlWithKey(expected: String, urlStr: String,
+ partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType)), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("jian", "http://j...@spark.apache.org/path?query=1;,
"USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
--- End diff --
Yes, you are right. Invalid `key` does this job. I'll fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69323801
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,43 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType)),
+expected)
+}
+def checkParseUrlWithKey(expected: String, urlStr: String,
+ partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType)), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("jian", "http://j...@spark.apache.org/path?query=1;,
"USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
--- End diff --
As invalid url and invalid part just get `null` result, I wonder in what
circumstance there would throw an exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69314031
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +657,125 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+
+ private lazy val stringExprs = children.toArray
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("parse_url function requires two or
three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ def parseUrlWithoutKey(url: Any, partToExtract: Any): Any = {
+if (url == null || partToExtract == null) {
+ null
+} else {
+ if (lastUrlStr == null || !url.equals(lastUrlStr)) {
--- End diff --
Thought we judge on the `url` string, the main purpose is to cache the
`URL` object.
As We must handle the exceptions caused by invalid urls, the approach of
`XPathBoolean` seems not suitable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69307883
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,43 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType)),
+expected)
+}
+def checkParseUrlWithKey(expected: String, urlStr: String,
+ partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType)), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("jian", "http://j...@spark.apache.org/path?query=1;,
"USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
--- End diff --
Oh sorry, I miss the point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69307503
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +657,125 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+
+ private lazy val stringExprs = children.toArray
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size > 3 || children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("parse_url function requires two or
three arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ def parseUrlWithoutKey(url: Any, partToExtract: Any): Any = {
+if (url == null || partToExtract == null) {
+ null
+} else {
+ if (lastUrlStr == null || !url.equals(lastUrlStr)) {
--- End diff --
Yes. When the `url` column has many same values.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69307232
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +657,125 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable = true
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+
+ private lazy val stringExprs = children.toArray
+ private val HOST = UTF8String.fromString("HOST")
+ private val PATH = UTF8String.fromString("PATH")
+ private val QUERY = UTF8String.fromString("QUERY")
+ private val REF = UTF8String.fromString("REF")
+ private val PROTOCOL = UTF8String.fromString("PROTOCOL")
+ private val FILE = UTF8String.fromString("FILE")
+ private val AUTHORITY = UTF8String.fromString("AUTHORITY")
+ private val USERINFO = UTF8String.fromString("USERINFO")
+ private val REGEXPREFIX = "(&|^)"
+ private val REGEXSUBFIX = "=([^&]*)"
--- End diff --
Make sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69307187
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +657,125 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
--- End diff --
OK. I'll open a JIRA for the codegen part and keep working on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69307094
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +657,125 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = """Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO.
+Key specifies which query to extract.
+Examples:
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'HOST')\n
'spark.apache.org'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY')\n
'query=1'
+ > SELECT _FUNC_('http://spark.apache.org/path?query=1', 'QUERY',
'query')\n '1'""")
+case class ParseUrl(children: Expression*)
--- End diff --
You are right. I'll fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69297640
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,43 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType)),
+expected)
+}
+def checkParseUrlWithKey(expected: String, urlStr: String,
+ partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType)), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("jian", "http://j...@spark.apache.org/path?query=1;,
"USERINFO")
+checkParseUrlWithKey("1", "http://spark.apache.org/path?query=1;,
"QUERY", "query")
+
+// Null checking
+checkParseUrl(null, null, "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, null)
+checkParseUrl(null, null, null)
+checkParseUrl(null, "test", "HOST")
+checkParseUrl(null, "http://spark.apache.org/path?query=1;, "NO")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"HOST", "query")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", "quer")
+checkParseUrlWithKey(null, "http://spark.apache.org/path?query=1;,
"QUERY", null)
--- End diff --
I am not sure. Is there any exceptional case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 @rxin and @dongjoon-hyun Thanks for your review. I have add a new commit which does following things: 1. Put `parse_url` function in the right order. 2. Use `""" """` instead of `+` in `extended` part to work with Scala 2.1. 3. Remove unnecessary `lazy`s. 4. Correct `REGEXPREFIX` and add a new null test case. 5. Use `NonFatal(_)` instead of the specified exception. 6. Fix the indentation problems. I have tried to not use varargs, but a separate constructor that accept two args does not help. As there isn't a magic key to make `parse_url(url, partToExtract, magic key)` to be treated as `parse_url(url, partToExtract)`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69276506
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
---
@@ -725,4 +725,43 @@ class StringExpressionsSuite extends SparkFunSuite
with ExpressionEvalHelper {
checkEvaluation(FindInSet(Literal("abf"), Literal("abc,b,ab,c,def")),
0)
checkEvaluation(FindInSet(Literal("ab,"), Literal("abc,b,ab,c,def")),
0)
}
+
+ test("ParseUrl") {
+def checkParseUrl(expected: String, urlStr: String, partToExtract:
String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType)),
+expected)
+}
+def checkParseUrlWithKey(expected: String, urlStr: String,
+ partToExtract: String, key: String): Unit = {
+ checkEvaluation(
+ParseUrl(Literal.create(urlStr, StringType),
Literal.create(partToExtract, StringType),
+ Literal.create(key, StringType)), expected)
+}
+
+checkParseUrl("spark.apache.org",
"http://spark.apache.org/path?query=1;, "HOST")
+checkParseUrl("/path", "http://spark.apache.org/path?query=1;, "PATH")
+checkParseUrl("query=1", "http://spark.apache.org/path?query=1;,
"QUERY")
+checkParseUrl("Ref", "http://spark.apache.org/path?query=1#Ref;, "REF")
+checkParseUrl("http", "http://spark.apache.org/path?query=1;,
"PROTOCOL")
+checkParseUrl("/path?query=1", "http://spark.apache.org/path?query=1;,
"FILE")
+checkParseUrl("spark.apache.org:8080",
"http://spark.apache.org:8080/path?query=1;, "AUTHORITY")
+checkParseUrl("jian", "http://j...@spark.apache.org/path?query=1;,
"USERINFO")
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69276469
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +655,128 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = "Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO\n"
+ + "key specifies which query to extract\n"
+ + "Examples:\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'HOST') FROM src LIMIT 1;\n" + " 'spark.apache.org'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY') FROM src LIMIT 1;\n" + " 'query=1'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY', 'query') FROM src LIMIT 1;\n" + " '1'")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ private lazy val stringExprs = children.toArray
--- End diff --
Try to avoid Scala Seqs' potential performance problems.
https://github.com/apache/spark/pull/13966/files#r69184719
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69264274
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +655,128 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = "Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO\n"
+ + "key specifies which query to extract\n"
+ + "Examples:\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'HOST') FROM src LIMIT 1;\n" + " 'spark.apache.org'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY') FROM src LIMIT 1;\n" + " 'query=1'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY', 'query') FROM src LIMIT 1;\n" + " '1'")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ private lazy val stringExprs = children.toArray
+ @transient private var lastUrlStr: UTF8String = _
+ @transient private var lastUrl: URL = _
+ // last key in string, we will update the pattern if key value changed.
+ @transient private var lastKey: UTF8String = _
+ // last regex pattern, we cache it for performance concern
+ @transient private var pattern: Pattern = _
+ private lazy val HOST: UTF8String = UTF8String.fromString("HOST")
--- End diff --
Yes, you are right. I'll fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69264025
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +655,128 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = "Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO\n"
+ + "key specifies which query to extract\n"
+ + "Examples:\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'HOST') FROM src LIMIT 1;\n" + " 'spark.apache.org'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY') FROM src LIMIT 1;\n" + " 'query=1'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY', 'query') FROM src LIMIT 1;\n" + " '1'")
+case class ParseUrl(children: Expression*)
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ override def nullable: Boolean = true
+
+ override def inputTypes: Seq[DataType] =
Seq.fill(children.size)(StringType)
+ override def dataType: DataType = StringType
+
+ private lazy val stringExprs = children.toArray
--- End diff --
This is for performance concern. I do not want to call `toArray` every row.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69263461
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +655,128 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = "Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO\n"
+ + "key specifies which query to extract\n"
+ + "Examples:\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'HOST') FROM src LIMIT 1;\n" + " 'spark.apache.org'\n"
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request: https://github.com/apache/spark/pull/14008#discussion_r69263441 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -653,6 +655,128 @@ case class StringRPad(str: Expression, len: Expression, pad: Expression) } /** + * Extracts a part from a URL + */ +@ExpressionDescription( + usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL", + extended = "Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE, USERINFO\n" + + "key specifies which query to extract\n" --- End diff -- Hi, @dongjoon-hyun . Thank you for review. I'll fix this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69251673
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -653,6 +655,128 @@ case class StringRPad(str: Expression, len:
Expression, pad: Expression)
}
/**
+ * Extracts a part from a URL
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(url, partToExtract[, key]) - extracts a part from a URL",
+ extended = "Parts: HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE,
USERINFO\n"
+ + "key specifies which query to extract\n"
+ + "Examples:\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'HOST') FROM src LIMIT 1;\n" + " 'spark.apache.org'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY') FROM src LIMIT 1;\n" + " 'query=1'\n"
+ + " > SELECT _FUNC_('http://spark.apache.org/path?query=1', "
+ + "'QUERY', 'query') FROM src LIMIT 1;\n" + " '1'")
+case class ParseUrl(children: Expression*)
--- End diff --
The parameter `key` can only be present when `partToExtract` is `QUERY`. So
there are two parameters call and three parameters call. I have not figured out
a better way to accomplish this.
I'll keep on trying to do it better. If there is any advice, that will be
very helpful. Thanks again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/14008#discussion_r69251415
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
---
@@ -285,6 +285,7 @@ object FunctionRegistry {
expression[StringTrimLeft]("ltrim"),
expression[JsonTuple]("json_tuple"),
expression[FormatString]("printf"),
+expression[ParseUrl]("parse_url"),
--- End diff --
OK, Thank you for review. I'll fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14008: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/14008 cc @rxin and @cloud-fan Improvements for performance concern --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14008: [SPARK-16281][SQL] Implement parse_url SQL functi...
GitHub user janplus opened a pull request: https://github.com/apache/spark/pull/14008 [SPARK-16281][SQL] Implement parse_url SQL function ## What changes were proposed in this pull request? This PR adds parse_url SQL functions in order to remove Hive fallback. A new implementation of #13999 ## How was this patch tested? Pass the exist tests including new testcases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/janplus/spark SPARK-16281 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14008.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14008 commit def598206e2bcfae4553edf18333872d2f2eff35 Author: wujian <jan.chou...@gmail.com> Date: 2016-07-01T05:24:34Z [SPARK-16281][SQL] Implement parse_url SQL function --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13999: [SPARK-16281][SQL] Implement parse_url SQL functi...
Github user janplus closed the pull request at: https://github.com/apache/spark/pull/13999 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13999: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/13999 This implementation is not that good. I'll close this one while create a new pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13966: [SPARK-16276][SQL] Implement elt SQL function
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13966#discussion_r69237086
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -162,6 +163,46 @@ case class ConcatWs(children: Seq[Expression])
}
}
+@ExpressionDescription(
+ usage = "_FUNC_(n, str1, str2, ...) - returns the n-th string, e.g.
returns str2 when n is 2",
+ extended = "> SELECT _FUNC_(1, 'scala', 'java') FROM src LIMIT 1;\n" +
"'scala'")
+case class Elt(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ private lazy val indexExpr = children.head
+ private lazy val stringExprs = children.tail.toArray
+
+ /** This expression is always nullable because it returns null if index
is out of range. */
+ override def nullable: Boolean = true
+
+ override def dataType: DataType = StringType
+
+ override def inputTypes: Seq[DataType] = IntegerType +:
Seq.fill(children.size - 1)(StringType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("elt function requires at least two
arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ override def eval(input: InternalRow): Any = {
+val indexObj = indexExpr.eval(input)
+if (indexObj == null) {
+ null
+} else {
+ val index = indexObj.asInstanceOf[Int]
+ if (index <= 0 || index > stringExprs.length) {
+null
+ } else {
+stringExprs(index - 1).eval(input)
--- End diff --
Oh, you are right. I'll fix this in my PR too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13999: [SPARK-16281][SQL] Implement parse_url SQL function
Github user janplus commented on the issue: https://github.com/apache/spark/pull/13999 cc @rxin and @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13999: [SPARK-16281][SQL] Implement parse_url SQL functi...
GitHub user janplus opened a pull request: https://github.com/apache/spark/pull/13999 [SPARK-16281][SQL] Implement parse_url SQL function ## What changes were proposed in this pull request? This PR adds `parse_url` SQL functions in order to remove Hive fallback. ## How was this patch tested? Pass the exist tests including new testcases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/janplus/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13999.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13999 commit 4e4d23d8f4da551a353eae2e071193943959b914 Author: wujian <jan.chou...@gmail.com> Date: 2016-06-30T17:15:11Z [SPARK-16281][SQL] Implement parse_url SQL function commit 5887e3ec589e8d12381c611f72123469bafdccda Author: wujian <jan.chou...@gmail.com> Date: 2016-06-30T17:16:48Z Merge remote-tracking branch 'upstream/master' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13966: [SPARK-16276][SQL] Implement elt SQL function
Github user janplus commented on a diff in the pull request:
https://github.com/apache/spark/pull/13966#discussion_r69098161
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
---
@@ -162,6 +163,46 @@ case class ConcatWs(children: Seq[Expression])
}
}
+@ExpressionDescription(
+ usage = "_FUNC_(n, str1, str2, ...) - returns the n-th string, e.g.
returns str2 when n is 2",
+ extended = "> SELECT _FUNC_(1, 'scala', 'java') FROM src LIMIT 1;\n" +
"'scala'")
+case class Elt(children: Seq[Expression])
+ extends Expression with ImplicitCastInputTypes with CodegenFallback {
+
+ private lazy val indexExpr = children.head
+ private lazy val stringExprs = children.tail.toArray
+
+ /** This expression is always nullable because it returns null if index
is out of range. */
+ override def nullable: Boolean = true
+
+ override def dataType: DataType = StringType
+
+ override def inputTypes: Seq[DataType] = IntegerType +:
Seq.fill(children.size - 1)(StringType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.size < 2) {
+ TypeCheckResult.TypeCheckFailure("elt function requires at least two
arguments")
+} else {
+ super[ImplicitCastInputTypes].checkInputDataTypes()
+}
+ }
+
+ override def eval(input: InternalRow): Any = {
+val indexObj = indexExpr.eval(input)
+if (indexObj == null) {
+ null
+} else {
+ val index = indexObj.asInstanceOf[Int]
+ if (index <= 0 || index > stringExprs.length) {
+null
+ } else {
+stringExprs(index - 1).eval(input)
--- End diff --
Why not just use children(index) instead of using an extra stringExprs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
