Apache Pinot Daily Email Digest (2022-03-23)

2022-03-23 Thread Pinot Slack Email Digest 
#general@paularmand.verhaegen: Hi all, I'm Paul-Armand Verhaegen. I'm the Data Domain and Data Specialty Architect for a News Publisher (we operate in a couple of European countries). Interested in basically anything with data (science and engineering), electronics, making stuff, math, hard problems, crypto tech, also organisational things related to data mesh. Here to learn about Pinot, what it can do for us, and which datasketches are useful in our RT dashboards.  @mark.needham: for sketches, have a look at a presentation Mayank did where he shows examples of where they work well -   @paularmand.verhaegen: Thanks. Watched it, was indeed an interesting presentation .@prashant.pandey: Hi team :slightly_smiling_face:. We have a use-case where we’d like to coalesce small segments to larger ones. However, it’s a realtime table and we use RT2OFF to move segments to offline servers periodically. Is it possible to use the minion merge rollup task to merge the segments residing on OFFLINE servers (although the docs explicitly mention that it only supports OFFLINE tables)? Thanks :slightly_smiling_face:  @mark.needham: Yeh - you can use RT2OFF to move segments to offline and then Merge Rollup to rollup the offline segments.

An example showing how to do the merge roll up here - 

Example of RT2OFF here -   @prashant.pandey: Thanks @mark.needham@asti.manuka: @asti.manuka has joined the channel@golthiryus: @golthiryus has joined the channel@krishna.gourishetti: @krishna.gourishetti has joined the channel#random@asti.manuka: @asti.manuka has joined the channel@golthiryus: @golthiryus has joined the channel@krishna.gourishetti: @krishna.gourishetti has joined the channel#troubleshooting@weixiang.sun: When I am trying to use the lookup UDF join between dimension table and realtime table, it does not work. But it works for dimension table and offline table, Is it expected? I do not see such restriction from . Is there anything missing?  @weixiang.sun: @lakshmanan.velusamy  @lakshmanan.velusamy: Getting the following exception :
```{
"errorCode": 200,
"message": "QueryExecutionError:\norg.apache.pinot.spi.exception.BadQueryRequestException: Caught exception while initializing transform function: lookup\n\tat org.apache.pinot.core.operator.transform.function.TransformFunctionFactory.get(TransformFunctionFactory.java:207)\n\tat org.apache.pinot.core.operator.transform.TransformOperator.(TransformOperator.java:56)\n\tat org.apache.pinot.core.plan.TransformPlanNode.run(TransformPlanNode.java:56)\n\tat org.apache.pinot.core.plan.SelectionPlanNode.run(SelectionPlanNode.java:83)\n\tat org.apache.pinot.core.plan.CombinePlanNode$1.callJob(CombinePlanNode.java:133)\n\tat org.apache.pinot.core.plan.CombinePlanNode$1.callJob(CombinePlanNode.java:119)\n\tat org.apache.pinot.core.util.trace.TraceCallable.call(TraceCallable.java:44)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)\n\tat shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)\n\tat shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)"
  },```
Query: `select LOOKUP(a, b, c, d) from table_REALTIME limit 10`  @weixiang.sun: @yupeng can you help us?@ryantle1028: hello everyone

i found some error whith transformFunction jsonPathString

i can not use word order in jsonPathString --> "transformFunction": "jsonPathString(order,'$.channel')"  -->this is not work.

i test modify json replace from order to hello

and user this --> "transformFunction": "jsonPathString(hello,'$.channel')"
it's working.  why i can not use "order". my real json massage  they use "order".

Please help.  @ryantle1028: Invalid transform function 'jsonPathString(order,'$.channel')' for column 'channel' exception: Invalid transform function 'jsonPathString(order,'$.channel')' for column 'channel'
Handled request from 172.23.188.107 POST , content-type application/json status code 400 Bad Request  @mark.needham: without looking into it any further yet, it might be b/c 'order' is a reserved word in sql and this is likely going through a sql parser...but maybe you can quote the word order and see if that works:

```"transformFunction": "jsonPathString(\"order\",'$.channel')"```  @ryantle1028: oh it's work thank very much @mark.needham  @ryantle1028: @mark.needham

"submissionDate" : "2022-03-15T17:31:44.540+0700"

how can i use date format with this?

I try to use this but not work

  "dateTimeFieldSpecs": [
{
  "name": "submissionDate",
  "da

Re: [VOTE] Apache Pinot 0.10.0 RC0

2022-03-23 Thread Xiang Fu 
+1(binding)

- verified sha512 hash
- verified signature
- verified git hash
- verified contents based on git commit hash & the downloaded source code
- verified LICENSE, NOTICE are correctly present
- ran quick start scripts

Thanks for putting up all those things together!


> On Mar 21, 2022, at 12:01 PM, Sajjad Moradi  wrote:
> 
> Hi Pinot Community,
> 
> This is a call for a vote to release Apache Pinot 0.10.0.
> 
> The release candidate:
> https://dist.apache.org/repos/dist/dev/pinot/apache-pinot-0.10.0-rc0/ 
> 
> 
> Git tag for this release:
> https://github.com/apache/pinot/tree/release-0.10.0-rc0 
> 
> 
> Git hash for this release:
> 30c4635bfeee88f88aa9c9f63b93bcd4a650607f
> 
> The artifact has been signed with key: 9079294B, which can be found in the 
> following KEYS file:
> https://dist.apache.org/repos/dist/release/pinot/KEYS 
> 
> 
> Release notes:
> https://github.com/apache/pinot/releases/tag/release-0.10.0-rc0 
> 
> 
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachepinot-1035 
> 
> 
> Documentation on verifying a release candidate:
> https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate
>  
> 
> 
> The vote will be open for at least 72 hours or until a necessary number of
> votes is reached.
> 
> Please vote accordingly,
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> Thanks,
> Apache Pinot team