Apache Pinot Daily Email Digest (2021-10-19)

Pinot Slack Email Digest Tue, 19 Oct 2021 19:00:30 -0700

#general

@piyush.chauhan: @piyush.chauhan has joined the channel
@chad: @chad has joined the channel
@r.sachdeva9355: @r.sachdeva9355 has joined the channel

#random

@piyush.chauhan: @piyush.chauhan has joined the channel
@chad: @chad has joined the channel
@r.sachdeva9355: @r.sachdeva9355 has joined the channel

#troubleshooting

@deemish2: Hi Team , there is a workaround to fix data gap issue temporarily - . Update watermark manually from where the data is available in realtime will move data from realtime table to offline table when the task is scheduled. Is it a feasible way to do this approach
@mayanks: Thanks @deemish2. You may want to add the details on how to update the watermark manually in the GH issue, for community members who might not know how to do so.
@jackie.jxt: I'd prefer handling it in the task scheduler. It will be hard to manage if we expect user to find the gap and manually bump up the watermark
@jackie.jxt: Will take a closer look at the code and see how we can handle it in the task scheduler. Shouldn't be hard since the data gap is handled within the merge-rollup task
@piyush.chauhan: @piyush.chauhan has joined the channel
@chad: @chad has joined the channel
@r.sachdeva9355: @r.sachdeva9355 has joined the channel

#pinot-dev

@xiangfu0: I’m seeing the kinesis integration test failed, seems the localstack doesn’t start ```Error: org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp Time elapsed: 175.147 s <<< FAILURE! cloud.localstack.docker.exception.LocalstackDockerException: Could not start the localstack docker container. at cloud.localstack.Localstack.startup(Localstack.java:97) at org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.startKinesis(RealtimeKinesisIntegrationTest.java:224) at org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp(RealtimeKinesisIntegrationTest.java:135) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108) at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224) at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146) at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:166) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105) at org.testng.TestRunner.privateRun(TestRunner.java:744) at org.testng.TestRunner.run(TestRunner.java:602) at org.testng.SuiteRunner.runTest(SuiteRunner.java:380) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340) at org.testng.SuiteRunner.run(SuiteRunner.java:289) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301) at org.testng.TestNG.runSuitesLocally(TestNG.java:1226) at org.testng.TestNG.runSuites(TestNG.java:1144) at org.testng.TestNG.run(TestNG.java:1115) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) Caused by: java.lang.IllegalStateException: Could not find token: Ready\. in Docker logs: 2021-10-19T08:14:44:DEBUG:plugin.manager: instantiating plugin PluginSpec(localstack.aws.provider.sts:default = <function sts at 0x7fa537678e60>)```
@kharekartik: Hi, yes I also noticed that today. Looking into it.
@kharekartik: It was working till yesterday so something changed in that time.
@xiangfu0: is there any change from github ci ? AlsoI cannot run it from my local laptop as well
@kharekartik: Testing in local. Will update you in few minutes.
@kharekartik: working fine in local. updating docker to latest version to see if that causes any issue.
@xiangfu0: I also updated the lib to 0.2.15, seems solved the issue

#announcements

@chad: @chad has joined the channel

#roadmap

@hardik.chheda: @hardik.chheda has joined the channel

#releases