#general
@zjureel: @zjureel has joined the channel
@mmadou: @mmadou has joined the channel
@mustafaf: @mustafaf has joined the channel
@helario: @helario has joined the channel
#random
@zjureel: @zjureel has joined the channel
@zjureel: @zjureel has left the channel
@mmadou: @mmadou has joined the channel
@mustafaf: @mustafaf has joined the channel
@helario: @helario has joined the channel
#troubleshooting
@zjureel: @zjureel has joined the channel
@mmadou: @mmadou has joined the channel
@arawat: ]
@deemish2: Hi All, If there is any gap in overlap data between realtime and offline time ( e.g. batch ingestion job crashed in offline table ) . In that case , is data available in hybrid table or not ?
@g.kishore: To handle such cases Pinot does max(time) - 1 day from offline. So as long as you rerun your batch ingestion job before the next days run, it will be fine
@mustafaf: @mustafaf has joined the channel
@tony: I am trying to configure the S3 storage to use server side encryption and have `pinot.controller.storage.factory.s3.serverSideEncryption=aws:kms` `pinot.controller.storage.factory.s3.ssekmsKeyId=KEY` and I get the rather odd error message ```Unknown value 'aws:kms' for S3PinotFS config: 'serverSideEncryption'. Supported values are: [AES256, aws:kms]```
@helario: @helario has joined the channel
@kchavda: Anyone using Pinot 0.8.0 and Superset? I upgraded Pinot to utilize the TIMESTAMP datatype but now Superset is giving me error when trying to create a dataset. Below is the error: ```Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: ERROR:root:'timestamp' Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: Traceback (most recent call last): Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/flask_appbuilder/api/__init__.py", line 84, in wraps Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return f(self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/views/base_api.py", line 80, in wraps Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: duration, response = time_function(f, self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/utils/core.py", line 1368, in time_function Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: response = func(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/utils/log.py", line 224, in wrapper Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: value = f(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/api.py", line 236, in post Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: new_model = CreateDatasetCommand(g.user, item).run() Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/commands/create.py", line 47, in run Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: self.validate() Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/commands/create.py", line 87, in validate Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: if database and not DatasetDAO.validate_table_exists( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/datasets/dao.py", line 81, in validate_table_exists Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: database.get_table(table_name, schema=schema) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/superset/superset/models/core.py", line 603, in get_table Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return Table( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "<string>", line 2, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return fn(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 560, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: metadata._remove_table(name, schema) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: compat.raise_( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: raise exception Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 555, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: table._init(name, metadata, *args, **kw) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 644, in _init Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: self._autoload( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 667, in _autoload Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: autoload_with.run_callable( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2212, in run_callable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return conn.run_callable(callable_, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1653, in run_callable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return callable_(self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 469, in reflecttable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return insp.reflecttable( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 664, in reflecttable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: for col_d in self.get_columns( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 390, in get_columns Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: col_defs = self.dialect.get_columns( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 390, in get_columns Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: columns = [ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 393, in <listcomp> Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: "type": get_type(spec["dataType"], spec.get("fieldSize")), Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 458, in get_type Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return type_map[data_type.lower()] Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: KeyError: 'timestamp'``` I've upgraded to latest pinotdb driver.
@richard892: Looks like a bug, can you submit a github bug report?
@kchavda: Hi @richard892, should I submit pinot or the python-pinot-dbapi repo?
@richard892: it looks like the python library needs updating to know about timestamps
@kchavda: Yeah, looks like the type_map doesn't have dates?
@richard892: submit the feature request to the python library
@kchavda: Just submitted issue.
@richard892: If nobody gets round to it I can submit a patch next week
#pinot-dev
@zjureel: @zjureel has joined the channel
#getting-started
@helario: @helario has joined the channel
@benshahbaz: @benshahbaz has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
