carloea2 commented on code in PR #8341:
URL: https://github.com/apache/texera/pull/8341#discussion_r3973978593
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala:
##########
@@ -145,4 +151,56 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
}
+ override def generateStandaloneCode(): String = {
+ // Strip to just the basename. The standalone script assumes the CSV
+ // lives in the same directory as the script (Texera's resolved URIs
+ // can't be used directly outside the system).
+ val basename = sourceBasename(fileName.getOrElse(""))
+
+ // Resolve the delimiter the same way the parser above does — first
character, empty
+ // means comma — and escape it. Every value the field accepts has to
survive this:
+ // pandas reads a separator longer than one character as a REGULAR
EXPRESSION, and a
+ // backslash spliced raw produced `sep="\"`, which is not valid Python at
all.
+ val sep =
customDelimiter.filter(_.nonEmpty).getOrElse(",").charAt(0).toString
+ // Texera's encoding enum uses values like UTF_8; pandas expects utf-8.
+ val encoding = fileEncoding.toString.replace("_", "-").toLowerCase
+ val headerArg = if (hasHeader) "0" else "None"
+
+ val args = scala.collection.mutable.ArrayBuffer[String]()
+ args += s"""filepath_or_buffer=${pyStringLiteral(basename)}"""
+ args += s"sep=${pyStringLiteral(sep)}"
+ args += s"""encoding=${pyStringLiteral(encoding)}"""
+ args += s"header=$headerArg"
+
+ // A CSV carries no types, so both readers infer, and they do not infer
+ // alike: the schema above tries TIMESTAMP and parses what it can, while
+ // pd.read_csv leaves a date column as text. Name the columns this operator
+ // decided were timestamps so pandas parses the same ones — by position
when
+ // there is no header, the frame's columns having no names until the rename
+ // below. A schema that cannot be read (an unresolved file) leaves the
+ // argument off rather than failing the export.
+ val dateColumns: Seq[String] =
+ Try(sourceSchema()).toOption.toSeq.flatMap(
+ _.getAttributes.zipWithIndex
+ .filter(_._1.getType == AttributeType.TIMESTAMP)
+ .map { case (a, i) => if (hasHeader) pyStringLiteral(a.getName) else
i.toString }
+ )
+ if (dateColumns.nonEmpty) args += s"parse_dates=[${dateColumns.mkString(",
")}]"
+
+ offset.foreach { o =>
+ // With a header, skip offset rows after row 0; without, skip offset
rows from the start.
+ if (hasHeader) args += s"skiprows=range(1, ${o + 1})"
+ else args += s"skiprows=$o"
+ }
+ limit.foreach(l => args += s"nrows=$l")
+
+ val readCall = s"out1df = pd.read_csv(${args.mkString(", ")})"
Review Comment:
pandas treats literal text such as "NA" as missing by default. With a CSV
text column containing NA and hello, Texera returns both strings, but the
export returns null and hello. I reproduced this through both execution paths.
Please match Texera's null parsing instead of using the pandas defaults.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala:
##########
@@ -145,4 +151,56 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
}
+ override def generateStandaloneCode(): String = {
+ // Strip to just the basename. The standalone script assumes the CSV
+ // lives in the same directory as the script (Texera's resolved URIs
+ // can't be used directly outside the system).
+ val basename = sourceBasename(fileName.getOrElse(""))
+
+ // Resolve the delimiter the same way the parser above does — first
character, empty
+ // means comma — and escape it. Every value the field accepts has to
survive this:
+ // pandas reads a separator longer than one character as a REGULAR
EXPRESSION, and a
+ // backslash spliced raw produced `sep="\"`, which is not valid Python at
all.
+ val sep =
customDelimiter.filter(_.nonEmpty).getOrElse(",").charAt(0).toString
+ // Texera's encoding enum uses values like UTF_8; pandas expects utf-8.
+ val encoding = fileEncoding.toString.replace("_", "-").toLowerCase
+ val headerArg = if (hasHeader) "0" else "None"
+
+ val args = scala.collection.mutable.ArrayBuffer[String]()
+ args += s"""filepath_or_buffer=${pyStringLiteral(basename)}"""
+ args += s"sep=${pyStringLiteral(sep)}"
+ args += s"""encoding=${pyStringLiteral(encoding)}"""
+ args += s"header=$headerArg"
+
+ // A CSV carries no types, so both readers infer, and they do not infer
+ // alike: the schema above tries TIMESTAMP and parses what it can, while
+ // pd.read_csv leaves a date column as text. Name the columns this operator
+ // decided were timestamps so pandas parses the same ones — by position
when
+ // there is no header, the frame's columns having no names until the rename
+ // below. A schema that cannot be read (an unresolved file) leaves the
+ // argument off rather than failing the export.
+ val dateColumns: Seq[String] =
+ Try(sourceSchema()).toOption.toSeq.flatMap(
+ _.getAttributes.zipWithIndex
+ .filter(_._1.getType == AttributeType.TIMESTAMP)
+ .map { case (a, i) => if (hasHeader) pyStringLiteral(a.getName) else
i.toString }
+ )
+ if (dateColumns.nonEmpty) args += s"parse_dates=[${dateColumns.mkString(",
")}]"
+
+ offset.foreach { o =>
+ // With a header, skip offset rows after row 0; without, skip offset
rows from the start.
+ if (hasHeader) args += s"skiprows=range(1, ${o + 1})"
+ else args += s"skiprows=$o"
+ }
+ limit.foreach(l => args += s"nrows=$l")
+
+ val readCall = s"out1df = pd.read_csv(${args.mkString(", ")})"
+
+ if (hasHeader) readCall
Review Comment:
Blank headers are renamed differently. For `id,,name` with row `1,2,alice`,
Texera exposes `column-2`, while the export exposes `Unnamed: 1`. A downstream
projection of column-2 then fails. Please apply the source schema's header
names to the exported frame.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/csv/CSVScanSourceOpDesc.scala:
##########
@@ -145,4 +151,56 @@ class CSVScanSourceOpDesc extends ScanSourceOpDesc {
}
+ override def generateStandaloneCode(): String = {
+ // Strip to just the basename. The standalone script assumes the CSV
+ // lives in the same directory as the script (Texera's resolved URIs
+ // can't be used directly outside the system).
+ val basename = sourceBasename(fileName.getOrElse(""))
+
+ // Resolve the delimiter the same way the parser above does — first
character, empty
+ // means comma — and escape it. Every value the field accepts has to
survive this:
+ // pandas reads a separator longer than one character as a REGULAR
EXPRESSION, and a
+ // backslash spliced raw produced `sep="\"`, which is not valid Python at
all.
+ val sep =
customDelimiter.filter(_.nonEmpty).getOrElse(",").charAt(0).toString
+ // Texera's encoding enum uses values like UTF_8; pandas expects utf-8.
+ val encoding = fileEncoding.toString.replace("_", "-").toLowerCase
+ val headerArg = if (hasHeader) "0" else "None"
+
+ val args = scala.collection.mutable.ArrayBuffer[String]()
+ args += s"""filepath_or_buffer=${pyStringLiteral(basename)}"""
+ args += s"sep=${pyStringLiteral(sep)}"
+ args += s"""encoding=${pyStringLiteral(encoding)}"""
+ args += s"header=$headerArg"
+
+ // A CSV carries no types, so both readers infer, and they do not infer
+ // alike: the schema above tries TIMESTAMP and parses what it can, while
+ // pd.read_csv leaves a date column as text. Name the columns this operator
+ // decided were timestamps so pandas parses the same ones — by position
when
+ // there is no header, the frame's columns having no names until the rename
+ // below. A schema that cannot be read (an unresolved file) leaves the
+ // argument off rather than failing the export.
+ val dateColumns: Seq[String] =
+ Try(sourceSchema()).toOption.toSeq.flatMap(
Review Comment:
The export endpoint passes fileName through without resolving it, so this
silently omits timestamp parsing for normal unresolved paths. I called the
resource with a valid local CSV path: Texera inferred TIMESTAMP, but the
exported script produced an object column of strings. Please resolve the source
before generating code, or pass its inferred schema into the export.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanOpDesc.scala:
##########
@@ -86,4 +91,63 @@ class FileScanOpDesc extends SourceOperatorDescriptor with
TextSourceOpDesc {
inputPorts = List(InputPort(displayName = "Filename")),
outputPorts = List(OutputPort())
)
+
+ override def generateStandaloneCode(): String = {
+ val col = attributeName
+ val enc = fileEncoding.toString.replace("_", "-").toLowerCase
+ val buf = scala.collection.mutable.ArrayBuffer[String]()
+
+ if (extract)
+ buf += "# WARNING: extract=true is not supported in standalone mode;
files are read as-is, not unpacked from archives."
+
+ val isBinary =
+ attributeType == FileAttributeType.BINARY || attributeType ==
FileAttributeType.LARGE_BINARY
+ val openArgs =
+ if (isBinary) """"rb""""
+ else s""""r", encoding=${pyStringLiteral(enc)}"""
+
+ buf += "_rows = []"
+ buf += "for _fn in in1df.iloc[:, 0]:"
Review Comment:
The native executor selects the first String field, not the first column.
With an input row containing integer id 99 followed by a valid file path,
Texera reads the file, while this export tries to open 99 and fails. Please use
the same filename selection rule.
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/fetcher/URLFetcherOpDesc.scala:
##########
@@ -87,4 +98,34 @@ class URLFetcherOpDesc extends SourceOperatorDescriptor {
outputPorts = List(OutputPort())
)
+ // The generated snippet uses `urllib.request`, which the translator's shared
+ // imports don't include. Following the per-operator convention (e.g. Split
+ // emits `import numpy as np`), the code block prepends its own import so the
+ // generated script is self-contained.
+ override def generateStandaloneCode(): String = {
+ val urlLiteral = objectMapper.writeValueAsString(url)
+ val isUtf8 = decodingMethod == DecodingMethod.UTF_8
+ val valueExpr = if (isUtf8) """_content.decode("utf-8")""" else "_content"
Review Comment:
Malformed UTF-8 is handled differently. A local HTTP server returning bytes
41 ff 42 produces `A�B` in Texera, but this export raises UnicodeDecodeError.
Normal responses and HTTP errors passed the same test. Please use replacement
decoding to match the native reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
