zyratlo commented on code in PR #8544:
URL: https://github.com/apache/texera/pull/8544#discussion_r4022292833
##########
frontend/src/app/workspace/component/notebook-import-modal/notebook-import-modal.component.html:
##########
@@ -17,69 +17,130 @@
under the License.
-->
-<div class="import-modal-diagram">
- <img
- ngSrc="assets/notebook_migration_tool/tool_popup_diagram.png"
- alt="Notebook to Workflow"
- width="1132"
- height="290" />
-</div>
-
<div class="import-modal-content">
<form
class="import-modal-form"
[formGroup]="importForm"
[attr.inert]="isSubmitting ? '' : null"
nz-form>
- <nz-form-item>
- <p class="import-modal-text">
- This tool converts a Python Jupyter Notebook into a Texera workflow
using LLM capabilities. After you submit a
- notebook, the LLM service generates a corresponding Texera workflow.
The conversion time depends on the
- notebook's complexity and can take 1-5 minutes. Once generation
finishes, you are taken to the new workflow,
- which opens with:
- </p>
- <ol class="import-modal-list">
- <li>
- The generated workflow ready to use (Note: you will still need to
upload the dataset and connect it to the
- workflow).
- </li>
- <li>A floating Jupyter window containing the uploaded notebook for
reference.</li>
- </ol>
- <p class="import-modal-text">
- Generation runs here after you submit. Please keep this window open
while you wait.
- </p>
- </nz-form-item>
+ <!-- One upload row shared by both tabs. They differ only in the accepted
extension and
+ their labels, so the markup lives here once and each tab passes its
own context. -->
+ <ng-template
+ #uploadRow
+ let-accept="accept"
+ let-label="label"
+ let-action="action">
+ <nz-form-item>
+ <nz-form-label [nzNoColon]="true">
+ <span class="import-modal-label">{{ label }}</span>
+ </nz-form-label>
+ <nz-form-control>
+ <div class="import-modal-upload-row">
+ <nz-upload
+ [nzAccept]="accept"
+ [nzBeforeUpload]="beforeUpload"
+ [nzShowUploadList]="false">
+ <button
+ nz-button
+ type="button"
+ [title]="action"
+ [attr.aria-label]="action">
+ <i
+ nz-icon
+ nzType="upload"></i>
+ </button>
+ </nz-upload>
- <nz-form-item>
- <nz-form-label [nzNoColon]="true">
- <span class="import-modal-label"> Upload Python Jupyter Notebook
</span>
- </nz-form-label>
- <nz-form-control>
- <div class="import-modal-upload-row">
- <nz-upload
- nzAccept=".ipynb"
- [nzBeforeUpload]="beforeUpload"
- [nzShowUploadList]="false">
- <button
- nz-button
- type="button"
- title="Upload notebook"
- aria-label="Upload notebook">
- <i
- nz-icon
- nzType="upload"></i>
- </button>
- </nz-upload>
+ <span
+ *ngIf="importForm.get('file')?.value?.name as fileName"
+ class="import-modal-selected-file"
+ [title]="fileName">
+ Selected file: {{ fileName }}
+ </span>
+ </div>
+ </nz-form-control>
+ </nz-form-item>
+ </ng-template>
- <span
- *ngIf="importForm.get('file')?.value?.name as fileName"
- class="import-modal-selected-file"
- [title]="fileName">
- Selected file: {{ fileName }}
- </span>
- </div>
- </nz-form-control>
- </nz-form-item>
+ <nz-tabs
+ [nzSelectedIndex]="selectedTabIndex"
+ [nzAnimated]="tabAnimation"
+ (nzSelectedIndexChange)="onTabChange($event)">
Review Comment:
There is already a video in the PR description covering the full flow,
including both tabs.
##########
frontend/src/app/workspace/service/notebook-migration/migration-prompts.ts:
##########
@@ -412,3 +412,173 @@ Here is an example of a mapping generated between the
given example Python code
}
Now create a mapping for the UDFs and the original code. Link the code blocks
marked by 'START <cell-uuid>' and 'END <cell-uuid>' with the UDF UUID's. The
code between them should be equivalent. Multiple cells can be mapped to the
same UDF when that UDF implements the logic of those cells. There could be any
number of cells and UDFs, so only create the correct number in the mapping.
Only give the mapping.
`;
+
+export const EXAMPLE_OF_MULTIPLE_UDF_CONVERSION_SCRIPT = `
+Here is an example of breaking up python code into multiple Texera UDFs.
Format your response structure exactly like the given example. The "code" key
contains a dictionary of the UDF ID's with their respective code. The "edges"
key contains a list of pairs that contains the connections between UDFs. The
"outputs" key contains a dictionary of the UDF ID's with a list of the output
column names of the DataFrame that the UDF yields. The UDFs can branch and
merge, it does not have to be a linear chain depending on your implementation.
+
+The original code is shown with each line prefixed by its line number and a
'|'. Those prefixes are annotations so that line ranges can be referred to
later. They are not part of the code and must never appear in the code you
generate.
+
+Original Code:
+\`\`\`python
+ 1| import pandas as pd
+ 2| from sklearn.model_selection import train_test_split
+ 3| from sklearn.ensemble import RandomForestClassifier
+ 4| from sklearn.svm import SVC
+ 5| from sklearn.tree import DecisionTreeClassifier
+ 6| from sklearn.linear_model import LogisticRegression
+ 7| from sklearn.metrics import accuracy_score
+ 8| from sklearn.preprocessing import StandardScaler
+ 9| import matplotlib.pyplot as plt
+10|
+11| # Load the dataset
+12| file_path = 'diabetes.csv'
+13| data = pd.read_csv(file_path)
+14|
+15| # Remove duplicate rows
+16| data = data.drop_duplicates()
+17|
+18| # Remove rows with null values
+19| data = data.dropna()
+20|
+21| # Print the minimum, maximum, and mean for all fields
+22| print("Minimum values:", data.min())
+23| print("Maximum values:", data.max())
+24| print("Mean values:", data.mean())
+25|
+26| # Create a boxplot for the 'Pregnancies' field
+27| plt.figure(figsize=(8, 6))
+28| plt.boxplot(data['Pregnancies'], vert=False, patch_artist=True)
+29| plt.title('Boxplot of Pregnancies')
+30| plt.xlabel('Number of Pregnancies')
+31| plt.show()
+32|
+33| # Separate features and target variable
+34| X = data.drop('Outcome', axis=1)
+35| y = data['Outcome']
+36|
+37| # Split data into training and testing sets (80% train, 20% test)
+38| X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
+39|
+40| scaler = StandardScaler()
+41| X_train = scaler.fit_transform(X_train)
+42| X_test = scaler.transform(X_test)
+43|
+44| # Train Random Forest model
+45| rf_model = RandomForestClassifier(random_state=42)
+46| rf_model.fit(X_train, y_train)
+47| rf_pred = rf_model.predict(X_test)
+48| rf_accuracy = accuracy_score(y_test, rf_pred)
+49| print(f"Random Forest Accuracy: {rf_accuracy:.2%}")
+50|
+51| # Train SVM model
+52| svm_model = SVC(random_state=42)
+53| svm_model.fit(X_train, y_train)
+54| svm_pred = svm_model.predict(X_test)
+55| svm_accuracy = accuracy_score(y_test, svm_pred)
+56| print(f"SVM Accuracy: {svm_accuracy:.2%}")
+\`\`\`
+
+Texera UDF conversion:
+\`\`\`json
+{
+ "code": {
+ "UDF1": "# UDF1\nfrom pytexera import *\nimport pandas as pd\nfrom
typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n # Remove duplicate rows\n data =
table.drop_duplicates()\n\n # Remove rows with null values\n data
= data.dropna()\n\n # Calculate statistics\n min_values =
data.min()\n max_values = data.max()\n mean_values =
data.mean()\n\n # Create a DataFrame to yield\n result_table =
pd.DataFrame({\n 'min_values': [min_values],\n
'max_values': [max_values],\n 'mean_values': [mean_values],\n
'data': [data]\n })\n\n yield Table(result_table)",
+ "UDF2": "# UDF2\nfrom pytexera import *\nimport pandas as pd\nimport
plotly.express as px\nimport plotly.io\nfrom typing import Iterator,
Optional\n\nclass ProcessTableOperator(UDFTableOperator):\n def
render_error(self, error_msg):\n return '''<h1>Boxplot is not
available.</h1>\n <p>Reason is: {} </p>\n
'''.format(error_msg)\n\n @overrides\n def process_table(self, table:
Table, port: int) -> Iterator[Optional[TableLike]]:\n data =
table['data'].iloc[0]\n\n if data.empty:\n yield
{'html-content': self.render_error('input table is empty.')}\n
return\n\n # Create a boxplot for the 'Pregnancies' field\n fig =
px.box(data, x='Pregnancies')\n fig.update_layout(margin=dict(l=0, r=0,
t=0, b=0))\n\n # Convert fig to HTML content\n html =
plotly.io.to_html(fig, include_plotlyjs='cdn', auto_play=False)\n yield
{'html-content': html}",
+ "UDF3": "# UDF3\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing
import StandardScaler\nfrom typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n data = table['data'].iloc[0]\n\n
# Separate features and target variable\n X = data.drop('Outcome',
axis=1)\n y = data['Outcome']\n\n # Split data into training and
testing sets (80% train, 20% test)\n X_train, X_test, y_train, y_test =
train_test_split(X, y, test_size=0.2, random_state=42)\n\n scaler =
StandardScaler()\n X_train = scaler.fit_transform(X_train)\n
X_test = scaler.transform(X_test)\n\n # Create a DataFrame to yield\n
result_table = pd.DataFrame({\n 'X_train': [X_train], 'X_test':
[X_test],\n 'y_train': [y_tra
in], 'y_test': [y_test]\n })\n\n yield Table(result_table)",
+ "UDF4": "# UDF4\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import
accuracy_score\nfrom typing import Iterator, Optional\n\nclass
ProcessTableOperator(UDFTableOperator):\n\n @overrides\n def
process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n X_train = table['X_train'].iloc[0]\n
y_train = table['y_train'].iloc[0]\n X_test =
table['X_test'].iloc[0]\n y_test = table['y_test'].iloc[0]\n\n #
Train Random Forest model\n rf_model =
RandomForestClassifier(random_state=42)\n rf_model.fit(X_train,
y_train)\n rf_pred = rf_model.predict(X_test)\n rf_accuracy =
accuracy_score(y_test, rf_pred)\n\n # Create a DataFrame to yield\n
result_table = pd.DataFrame({\n 'rf_model': [rf_model],\n
'rf_accuracy': [rf_accuracy],\n 'X_test': [X_test],\n
'y_test': [y_test]\n
})\n\n yield Table(result_table)",
+ "UDF5": "# UDF5\nfrom pytexera import *\nimport pandas as pd\nfrom
sklearn.svm import SVC\nfrom sklearn.metrics import accuracy_score\nfrom typing
import Iterator, Optional\n\nclass ProcessTableOperator(UDFTableOperator):\n\n
@overrides\n def process_table(self, table: Table, port: int) ->
Iterator[Optional[TableLike]]:\n X_train = table['X_train'].iloc[0]\n
y_train = table['y_train'].iloc[0]\n X_test =
table['X_test'].iloc[0]\n y_test = table['y_test'].iloc[0]\n\n #
Train SVM model\n svm_model = SVC(random_state=42)\n
svm_model.fit(X_train, y_train)\n svm_pred = svm_model.predict(X_test)\n
svm_accuracy = accuracy_score(y_test, svm_pred)\n\n # Create a
DataFrame to yield\n result_table = pd.DataFrame({\n
'svm_model': [svm_model],\n 'svm_accuracy': [svm_accuracy],\n
'X_test': [X_test],\n 'y_test': [y_test]\n })\n\n
yield Table(result_tab
le)"
+ },
+ "edges": [
+ ["UDF1", "UDF2"],
+ ["UDF1", "UDF3"],
+ ["UDF3", "UDF4"],
+ ["UDF3", "UDF5"]
+ ],
+ "outputs": {
+ "UDF1": ["min_values", "max_values", "mean_values", "data"],
+ "UDF2": ["html-content"],
+ "UDF3": ["X_train", "X_test", "y_train", "y_test"],
+ "UDF4": ["rf_model", "rf_accuracy", "X_test", "y_test"],
+ "UDF5": ["svm_model", "svm_accuracy", "X_test", "y_test"]
+ }
+}
+\`\`\``;
+
+export const SCRIPT_WORKFLOW_PROMPT = `You are an expert in Python coding and
workflow systems.
+Many users of Texera system are non-technical, but the Python scripts they
provide are written by technical people.
+They want to convert their scripts to Texera workflows.
+Your goal is to help convert these scripts into a Texera workflow that
non-technical users can use directly.
+So do not remove or modify any classes or functions, preserve their names and
structure as they are.
+Ensure that all essential logic remains intact.
+Create multiple Texera UDF codes using the provided Python code.
+Number each UDF, starting at 1 and incrementing, by starting with a comment
that states that UDF number.
+
+Use the class and function names as shown in ProcessTupleOperator,
ProcessTableOperator, and ProcessBatchOperator.
+Do not change the class names, function names, or input parameters.
+Use the ones that make sense and split the code meaningfully as instructed.
+
+Use the starter code provided for Python UDFs.
+
+Use the documentation of Table, Tuple, or Batch to work with parameters within
Texera UDF.
+Do not import other libraries to define these types.
+
+There is no need for an __init__ function. Assume all inputs are valid pandas
DataFrames,
+so do not use .to_pandas(), .to_dataframe(), etc. Do not load data from a file
in the first UDF;
+the workflow's source operator supplies the initial data, so assume it is
already given to you in the
+table parameter. Replacing file-loading code with this input is the one
exception to preserving all
+original code (see below).
+Ensure proper data flow between functions. Separate operators as if they will
run in different files.
+
+Current UDF operators can only have one output. Build a dataframe to yield all
necessary variables
+and data. Ensure proper data flow for each UDF and all information is yielded
(including training
+and testing data) if subsequent UDFs need them.
+
+Ensure all necessary imports are included in each UDF code block.
+
+Each UDF operator should be in its own Python code block. Do not combine them
into a single block.
+Ensure import statements cover all used functions and separate them as
necessary.
+
+It is VERY important that all of the original code in the Python script is
represented in the generated workflow.
+Make sure that nothing in the original is removed and that the semantic
meaning of what the original code was doing is retained.
+The only exception is data-loading code (e.g. pd.read_csv); it is represented
by the workflow's input/source operator rather than copied into a UDF.
+If there are user-defined Python classes, include the entire class definition
in the appropriate UDF(s) that use that class.
+Always include the code that defines the class inside of every distinct UDF
that uses that constructs an object of that class.
Review Comment:
Fixed in 242e495. The line now reads "Always include the full class
definition inside every UDF that references that class, including every UDF
that constructs an object of it." I kept the reference case rather than
narrowing to construction as suggested, since a UDF that receives an instance
and calls a method on it also needs the definition for unpickling. The same
malformed sentence was in the notebook WORKFLOW_PROMPT, so both copies are
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
