[incubator-superset] branch master updated: [sqllab] Add CUSTOM_TEMPLATE_PROCESSOR config (#9376)
This is an automated email from the ASF dual-hosted git repository.
dpgaspar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-superset.git
The following commit(s) were added to refs/heads/master by this push:
new 72f051f [sqllab] Add CUSTOM_TEMPLATE_PROCESSOR config (#9376)
72f051f is described below
commit 72f051f3ce6b269a0d80de0cef3f357826142213
Author: dandanhub
AuthorDate: Tue Apr 7 13:00:42 2020 -0700
[sqllab] Add CUSTOM_TEMPLATE_PROCESSOR config (#9376)
Co-authored-by: Dandan Shi
---
docs/installation.rst | 53 +++
docs/sqllab.rst | 9 +++
superset/config.py| 10 +++
superset/extensions.py| 15 +++-
superset/jinja_context.py | 4 +-
tests/base_tests.py | 22 ++
tests/core_tests.py | 83 +++
tests/superset_test_config.py | 5 ++
tests/superset_test_custom_template_processors.py | 59
9 files changed, 258 insertions(+), 2 deletions(-)
diff --git a/docs/installation.rst b/docs/installation.rst
index 84dd6e1..3528057 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -1087,6 +1087,59 @@ in this dictionary are made available for users to use
in their SQL.
'my_crazy_macro': lambda x: x*2,
}
+Besides default Jinja templating, SQL lab also supports self-defined template
+processor by setting the ``CUSTOM_TEMPLATE_PROCESSORS`` in your superset
configuration.
+The values in this dictionary overwrite the default Jinja template processors
of the
+specified database engine.
+The example below configures a custom presto template processor which
implements
+its own logic of processing macro template with regex parsing. It uses ``$``
style
+macro instead of ``{{ }}`` style in Jinja templating. By configuring it with
+``CUSTOM_TEMPLATE_PROCESSORS``, sql template on presto database is processed
+by the custom one rather than the default one.
+
+.. code-block:: python
+
+def DATE(
+ts: datetime, day_offset: SupportsInt = 0, hour_offset: SupportsInt = 0
+) -> str:
+"""Current day as a string."""
+day_offset, hour_offset = int(day_offset), int(hour_offset)
+offset_day = (ts + timedelta(days=day_offset,
hours=hour_offset)).date()
+return str(offset_day)
+
+class CustomPrestoTemplateProcessor(PrestoTemplateProcessor):
+"""A custom presto template processor."""
+
+engine = "presto"
+
+def process_template(self, sql: str, **kwargs) -> str:
+"""Processes a sql template with $ style macro using regex."""
+# Add custom macros functions.
+macros = {
+"DATE": partial(DATE, datetime.utcnow())
+} # type: Dict[str, Any]
+# Update with macros defined in context and kwargs.
+macros.update(self.context)
+macros.update(kwargs)
+
+def replacer(match):
+"""Expand $ style macros with corresponding function calls."""
+macro_name, args_str = match.groups()
+args = [a.strip() for a in args_str.split(",")]
+if args == [""]:
+args = []
+f = macros[macro_name[1:]]
+return f(*args)
+
+macro_names = ["$" + name for name in macros.keys()]
+pattern = r"(%s)\s*\(([^()]*)\)" % "|".join(map(re.escape,
macro_names))
+return re.sub(pattern, replacer, sql)
+
+CUSTOM_TEMPLATE_PROCESSORS = {
+CustomPrestoTemplateProcessor.engine: CustomPrestoTemplateProcessor
+}
+
+
SQL Lab also includes a live query validation feature with pluggable backends.
You can configure which validation implementation is used with which database
engine by adding a block like the following to your config.py:
diff --git a/docs/sqllab.rst b/docs/sqllab.rst
index 992a689..aace28f 100644
--- a/docs/sqllab.rst
+++ b/docs/sqllab.rst
@@ -104,6 +104,15 @@ environment using the configuration variable
``JINJA_CONTEXT_ADDONS``.
All objects referenced in this dictionary will become available for users
to integrate in their queries in **SQL Lab**.
+Customize templating
+
+
+As mentioned in the `Installation & Configuration
`__
documentation,
+it's possible for administrators to overwrite Jinja templating with your
customized
+template processor using the configuration variable
``CUSTOM_TEMPLATE_PROCESSORS``.
+The template processors referenced in the dictionary will overwrite default
Jinja template processors
+of the specified database engines.
+
Query cost estimation
'
diff --git a/superset/config.py b/superset/config.py
index 1cef75f..1d271
[incubator-superset] branch master updated: Handle empty dataframes in TableViz (#9480)
This is an automated email from the ASF dual-hosted git repository.
maximebeauchemin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-superset.git
The following commit(s) were added to refs/heads/master by this push:
new a52b9ee Handle empty dataframes in TableViz (#9480)
a52b9ee is described below
commit a52b9ee8fffb123d98e111c26ca82714deae49de
Author: Luca Toscano
AuthorDate: Tue Apr 7 19:37:03 2020 +0200
Handle empty dataframes in TableViz (#9480)
TableViz fails to display empty dataframes returning an error like:
"None of [Index(['project', 'count'], dtype='object')] are in the [columns]"
The behavior has been observed while testing 0.36.0rc3 with
Druid datasources.
issue: #9468
---
superset/viz.py | 23 ---
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/superset/viz.py b/superset/viz.py
index 5a9a6ac..f17f234 100644
--- a/superset/viz.py
+++ b/superset/viz.py
@@ -625,17 +625,18 @@ class TableViz(BaseViz):
self.form_data.get("percent_metrics") or []
)
-df = pd.concat(
-[
-df[non_percent_metric_columns],
-(
-df[percent_metric_columns]
-.div(df[percent_metric_columns].sum())
-.add_prefix("%")
-),
-],
-axis=1,
-)
+if not df.empty:
+df = pd.concat(
+[
+df[non_percent_metric_columns],
+(
+df[percent_metric_columns]
+.div(df[percent_metric_columns].sum())
+.add_prefix("%")
+),
+],
+axis=1,
+)
data = self.handle_js_int_overflow(
dict(records=df.to_dict(orient="records"),
columns=list(df.columns))
[incubator-superset] branch master updated: Filter owners select by text input (#9337)
This is an automated email from the ASF dual-hosted git repository.
erikrit pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-superset.git
The following commit(s) were added to refs/heads/master by this push:
new 5e53506 Filter owners select by text input (#9337)
5e53506 is described below
commit 5e535062da95135b133f9b924fe93501ae6fcc41
Author: David Aaron Suddjian <1858430+suddj...@users.noreply.github.com>
AuthorDate: Tue Apr 7 09:09:02 2020 -0700
Filter owners select by text input (#9337)
* filter owners select by text input
* use rison
* fix backend owners filter logic
* use fullname, not username on owners inputs
* fix some tests
* fixing tests
* deterministic tests
* appease linter
* add back search by username
* more comprehensive filter test
* add clarifying text
* formatting...
---
superset-frontend/package-lock.json| 290 ++---
superset-frontend/package.json | 4 +-
.../src/dashboard/components/PropertiesModal.jsx | 130 +
.../src/explore/components/PropertiesModal.tsx | 63 +++--
superset/charts/api.py | 9 +-
superset/dashboards/api.py | 9 +-
superset/datasets/api.py | 11 +-
superset/views/base_api.py | 35 ++-
superset/views/filters.py | 48
tests/base_api_tests.py| 18 +-
tests/chart_api_tests.py | 9 +-
tests/dashboards/api_tests.py | 9 +-
12 files changed, 366 insertions(+), 269 deletions(-)
diff --git a/superset-frontend/package-lock.json
b/superset-frontend/package-lock.json
index 2c64cf2..81cf62a 100644
--- a/superset-frontend/package-lock.json
+++ b/superset-frontend/package-lock.json
@@ -8795,14 +8795,11 @@
}
},
"@types/react-select": {
- "version": "3.0.10",
- "resolved":
"https://registry.npmjs.org/@types/react-select/-/react-select-3.0.10.tgz";,
- "integrity":
"sha512-oUHXqvbkRhC07q5JjeY6hE+NUqgUM6CyaRXEKYPvMCBqUOuLnYltyhiNx6Jpb+iFpYtNHSQtF4dNJfMdMooKoQ==",
- "dev": true,
+ "version": "1.3.4",
+ "resolved":
"https://registry.npmjs.org/@types/react-select/-/react-select-1.3.4.tgz";,
+ "integrity":
"sha512-0BwjswNzKBszG5O4xq72W54NrrbmOZvJfaM/Dwru3F6DhvFO9nihMP1IRzXSOJ1qGRCS3VCu9FnBYJ+25lSldw==",
"requires": {
-"@types/react": "*",
-"@types/react-dom": "*",
-"@types/react-transition-group": "*"
+"@types/react": "*"
}
},
"@types/react-table": {
@@ -8814,15 +8811,6 @@
"@types/react": "*"
}
},
-"@types/react-transition-group": {
- "version": "4.2.4",
- "resolved":
"https://registry.npmjs.org/@types/react-transition-group/-/react-transition-group-4.2.4.tgz";,
- "integrity":
"sha512-8DMUaDqh0S70TjkqU0DxOu80tFUiiaS9rxkWip/nb7gtvAsbqOXm02UCmR8zdcjWujgeYPiPNTVpVpKzUDotwA==",
- "dev": true,
- "requires": {
-"@types/react": "*"
- }
-},
"@types/react-ultimate-pagination": {
"version": "1.2.0",
"resolved":
"https://registry.npmjs.org/@types/react-ultimate-pagination/-/react-ultimate-pagination-1.2.0.tgz";,
@@ -8849,6 +8837,11 @@
"@types/react": "*"
}
},
+"@types/rison": {
+ "version": "0.0.6",
+ "resolved": "https://registry.npmjs.org/@types/rison/-/rison-0.0.6.tgz";,
+ "integrity":
"sha512-mE3eRK0fpTN/GnNBOIg2tGq2cFhchQXF6fCbrLxus75TgnoOECbdHikr948FGO/UAml7/ZhLMa5FbGkF5PKvmw=="
+},
"@types/shortid": {
"version": "0.0.29",
"resolved":
"https://registry.npmjs.org/@types/shortid/-/shortid-0.0.29.tgz";,
@@ -11512,28 +11505,28 @@
"dependencies": {
"abbrev": {
"version": "1.1.1",
- "resolved": false,
+ "resolved": "",
"integrity":
"sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q==",
"dev": true,
"optional": true
},
"ansi-regex": {
"version": "2.1.1",
- "resolved": false,
+ "resolved": "",
"integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=",
"dev": true,
"optional": true
},
"aproba": {
"version": "1.2.0",
- "resolved": false,
+ "resolved": "",
"integrity":
"sha512-Y9J6ZjXtoYh8RnXVCMOU/ttDmk1aBjunq9vO0ta5x85WDQiQfUF9sIPBITdbiiIVcBo03Hi3jMxigBtsddlXRw==",
"dev": true,
"optional": true
},
"are-we-there-yet": {
"version": "1.1.5",
- "resolved": false,
+ "res
[incubator-superset] branch master updated (f9db3fa -> 4be8275)
This is an automated email from the ASF dual-hosted git repository. dpgaspar pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-superset.git. from f9db3fa [mypy] Enforcing typing for superset.dashboards (#9418) add 4be8275 [datasets] Add strict type annotation (#9437) No new revisions were added by this update. Summary of changes: setup.cfg| 2 +- superset/datasets/api.py | 3 ++- superset/datasets/commands/create.py | 3 ++- superset/datasets/commands/delete.py | 3 ++- superset/datasets/commands/exceptions.py | 22 +++--- superset/datasets/commands/refresh.py| 18 ++ superset/datasets/commands/update.py | 27 +-- superset/datasets/dao.py | 16 ++-- superset/datasets/schemas.py | 2 +- superset/views/base.py | 2 +- 10 files changed, 57 insertions(+), 41 deletions(-)
[incubator-superset] branch master updated: [mypy] Enforcing typing for superset.dashboards (#9418)
This is an automated email from the ASF dual-hosted git repository. dpgaspar pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-superset.git The following commit(s) were added to refs/heads/master by this push: new f9db3fa [mypy] Enforcing typing for superset.dashboards (#9418) f9db3fa is described below commit f9db3faadec5e651fd3872b7af21f4099359ea06 Author: Daniel Vaz Gaspar AuthorDate: Tue Apr 7 12:52:14 2020 +0100 [mypy] Enforcing typing for superset.dashboards (#9418) * [mypy] Enforcing typing for superset.dashboards * Make return types equal on all commands * Make return types equal on all commands * [dashboard] address comments same return type on commands * lint * lint --- setup.cfg | 2 +- superset/charts/commands/delete.py | 4 ++-- superset/charts/commands/update.py | 4 ++-- superset/commands/base.py | 6 ++ superset/dashboards/api.py | 7 +-- superset/dashboards/commands/bulk_delete.py | 3 ++- superset/dashboards/commands/create.py | 3 ++- superset/dashboards/commands/delete.py | 3 ++- superset/dashboards/commands/exceptions.py | 2 +- superset/dashboards/commands/update.py | 7 --- superset/dashboards/dao.py | 13 +++-- superset/dashboards/filters.py | 5 - superset/dashboards/schemas.py | 7 --- superset/views/base.py | 8 +--- 14 files changed, 43 insertions(+), 31 deletions(-) diff --git a/setup.cfg b/setup.cfg index 33e7223..bff4adf 100644 --- a/setup.cfg +++ b/setup.cfg @@ -53,7 +53,7 @@ order_by_type = false ignore_missing_imports = true no_implicit_optional = true -[mypy-superset.bin.*,superset.charts.*,superset.commands.*,superset.common.*,superset.dao.*,superset.db_engine_specs.*,superset.db_engines.*,superset.examples.*] +[mypy-superset.bin.*,superset.charts.*,superset.dashboards.*,superset.commands.*,superset.common.*,superset.dao.*,superset.db_engine_specs.*,superset.db_engines.*,superset.examples.*] check_untyped_defs = true disallow_untyped_calls = true disallow_untyped_defs = true diff --git a/superset/charts/commands/delete.py b/superset/charts/commands/delete.py index bf85d9e..c21abee 100644 --- a/superset/charts/commands/delete.py +++ b/superset/charts/commands/delete.py @@ -27,9 +27,9 @@ from superset.charts.commands.exceptions import ( ) from superset.charts.dao import ChartDAO from superset.commands.base import BaseCommand -from superset.connectors.sqla.models import SqlaTable from superset.dao.exceptions import DAODeleteFailedError from superset.exceptions import SupersetSecurityException +from superset.models.slice import Slice from superset.views.base import check_ownership logger = logging.getLogger(__name__) @@ -39,7 +39,7 @@ class DeleteChartCommand(BaseCommand): def __init__(self, user: User, model_id: int): self._actor = user self._model_id = model_id -self._model: Optional[SqlaTable] = None +self._model: Optional[Slice] = None def run(self) -> Model: self.validate() diff --git a/superset/charts/commands/update.py b/superset/charts/commands/update.py index 5d9ee2a..6a110d5 100644 --- a/superset/charts/commands/update.py +++ b/superset/charts/commands/update.py @@ -32,10 +32,10 @@ from superset.charts.commands.exceptions import ( from superset.charts.dao import ChartDAO from superset.commands.base import BaseCommand from superset.commands.utils import get_datasource_by_id, populate_owners -from superset.connectors.sqla.models import SqlaTable from superset.dao.exceptions import DAOUpdateFailedError from superset.dashboards.dao import DashboardDAO from superset.exceptions import SupersetSecurityException +from superset.models.slice import Slice from superset.views.base import check_ownership logger = logging.getLogger(__name__) @@ -46,7 +46,7 @@ class UpdateChartCommand(BaseCommand): self._actor = user self._model_id = model_id self._properties = data.copy() -self._model: Optional[SqlaTable] = None +self._model: Optional[Slice] = None def run(self) -> Model: self.validate() diff --git a/superset/commands/base.py b/superset/commands/base.py index 9c6de0c..998a756 100644 --- a/superset/commands/base.py +++ b/superset/commands/base.py @@ -15,9 +15,7 @@ # specific language governing permissions and limitations # under the License. from abc import ABC, abstractmethod -from typing import Optional - -from flask_appbuilder.models.sqla import Model +from typing import Any class BaseCommand(ABC): @@ -26,7 +24,7 @@ class BaseCommand(ABC): """ @abstractmethod -def run(self) -> Optional[Model]: +def run(self) -> Any: """ Run executes the command. Can raise command excepti
[incubator-superset] branch master updated (b6bca9f -> b487834)
This is an automated email from the ASF dual-hosted git repository. villebro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-superset.git. from b6bca9f Migrating shared DeckGL controls (#9455) add b487834 [Doc] Update installation doc for Dremio (#9464) No new revisions were added by this update. Summary of changes: docs/installation.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
