Apache Pinot Daily Email Digest (2021-04-15)

Pinot Slack Email Digest Thu, 15 Apr 2021 19:00:37 -0700

#general

@raj: @raj has joined the channel
@kuantian.zhang01: @kuantian.zhang01 has joined the channel
@selvakumarcts: @selvakumarcts has joined the channel
@kautsshukla: @kautsshukla has joined the channel
@roniepaolo: @roniepaolo has joined the channel
@grebaza: @grebaza has joined the channel
@neilteng233: Hey, I have a question about the pinot-stream-ingestion. What will happen to the data after flush? Will it be consolidated into disk or just removed?
@mayanks: Data is flushed to disk and still used for serving until it is older than the retention period, after which it is removed.
@neilteng233: OK. So I also configure the retention period in the tabl config JSON? what is the key/name of the retention config?
@mayanks:
@mayanks: Grep for retention
@neilteng233: Got you thank you!
@btripathy: @btripathy has joined the channel
@miliang: Hey, build on master failed with message: ```[INFO] Reactor Summary for Pinot 0.8.0-SNAPSHOT: [INFO] [INFO] Pinot .............................................. SUCCESS [ 15.773 s] [INFO] Pinot Service Provider Interface ................... SUCCESS [ 2.762 s] [INFO] Pinot Segment Service Provider Interface ........... SUCCESS [ 1.093 s] [INFO] Pinot Plugins ...................................... SUCCESS [ 4.414 s] [INFO] Pinot Metrics ...................................... SUCCESS [ 0.145 s] [INFO] Pinot Yammer Metrics ............................... SUCCESS [ 4.240 s] [INFO] Pinot Common ....................................... SUCCESS [ 16.085 s] [INFO] Pinot Input Format ................................. SUCCESS [ 0.993 s] [INFO] Pinot Avro Base .................................... SUCCESS [ 1.109 s] [INFO] Pinot Avro ......................................... SUCCESS [ 0.902 s] [INFO] Pinot Csv .......................................... SUCCESS [ 0.354 s] [INFO] Pinot JSON ......................................... SUCCESS [ 0.344 s] [INFO] Pinot local segment implementations ................ SUCCESS [ 6.542 s] [INFO] Pinot Core ......................................... SUCCESS [ 9.313 s] [INFO] Pinot Server ....................................... SUCCESS [ 4.548 s] [INFO] Pinot Segment Uploader ............................. SUCCESS [ 1.528 s] [INFO] Pinot Segment Uploader Default ..................... SUCCESS [ 15.425 s] [INFO] Pinot Controller ................................... SUCCESS [ 50.838 s] [INFO] Pinot Broker ....................................... SUCCESS [ 4.736 s] [INFO] Pinot Clients ...................................... SUCCESS [ 0.117 s] [INFO] Pinot Java Client .................................. SUCCESS [ 0.474 s] [INFO] Pinot JDBC Client .................................. SUCCESS [ 0.555 s] [INFO] Pinot Batch Ingestion .............................. SUCCESS [ 1.501 s] [INFO] Pinot Batch Ingestion Common ....................... SUCCESS [ 0.393 s] [INFO] Pinot Minion ....................................... SUCCESS [ 1.718 s] [INFO] Pinot Confluent Avro ............................... FAILURE [ 0.586 s] [INFO] Pinot ORC .......................................... SKIPPED [INFO] Pinot Parquet ...................................... SKIPPED [INFO] Pinot Thrift ....................................... SKIPPED [INFO] Pinot Protocol Buffers ............................. SKIPPED [INFO] Pluggable Pinot file system ........................ SKIPPED [INFO] Pinot Azure Data Lake Storage ...................... SKIPPED [INFO] Pinot Hadoop Filesystem ............................ SKIPPED [INFO] Pinot Google Cloud Storage ......................... SKIPPED [INFO] Pinot Amazon S3 .................................... SKIPPED [INFO] Pinot Batch Ingestion for Spark .................... SKIPPED [INFO] Pinot Batch Ingestion for Hadoop ................... SKIPPED [INFO] Pinot Batch Ingestion Standalone ................... SKIPPED [INFO] Pinot Batch Ingestion .............................. SKIPPED [INFO] Pinot Ingestion Common ............................. SKIPPED [INFO] Pinot Hadoop ....................................... SKIPPED [INFO] Pinot Spark ........................................ SKIPPED [INFO] Pinot Stream Ingestion ............................. SKIPPED [INFO] Pinot Kafka Base ................................... SKIPPED [INFO] Pinot Kafka 0.9 .................................... SKIPPED [INFO] Pinot Kafka 2.0 .................................... SKIPPED [INFO] Pinot Minion Tasks ................................. SKIPPED [INFO] Pinot Minion Built-In Tasks ........................ SKIPPED [INFO] Pinot Segment Writer ............................... SKIPPED [INFO] Pinot Segment Writer File Based .................... SKIPPED [INFO] Pinot Tools ........................................ SKIPPED [INFO] Pinot Integration Tests ............................ SKIPPED [INFO] Pinot Perf ......................................... SKIPPED [INFO] Pinot Distribution ................................. SKIPPED [INFO] Pinot Connectors ................................... SKIPPED [INFO] Pinot Spark Connector .............................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:26 min [INFO] Finished at: 2021-04-15T15:49:12-07:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project pinot-confluent-avro: Could not resolve dependencies for project org.apache.pinot:pinot-confluent-avro:jar:0.8.0-SNAPSHOT: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.3.1: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.3.1: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:5.3.1 from/to maven-default-http-blocker (): Blocked mirror for repositories: [confluent (, default, releases+snapshots)] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :pinot-confluent-avro ➜ incubator-pinot git:(master)```
@steotia: Move this to pinot-dev ?
@steotia: <#C013WKLT5T7|pinot-dev> channel
@jlli: or <#C011C9JHN7R|troubleshooting> channel
@miliang: ```Failed to execute goal on project pinot-confluent-avro: Could not resolve dependencies for project org.apache.pinot:pinot-confluent-avro:jar:0.8.0-SNAPSHOT: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.3.1: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.3.1: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:5.3.1 from/to maven-default-http-blocker (): Blocked mirror for repositories: [confluent (, default, releases+snapshots)] -> [Help 1] [ERROR]```
@btripathy: Hello guys..I am new to Pinot and going through the architecture . Got a question ,what will be the format of the segments in the deep storage if kept on S3 ?
@dlavoie: Pinot format
@dlavoie: Basically a copy of what is hosted by the servers when the segments are live
@btripathy: :thumbsup:
@fx19880617: also it’s tar.gz’ed
@gabuglc: Hello guys, what is the correct value for "stream.kafka.decoder.class.name" when decoding an avro message from Schema Registry?
@fx19880617:
@fx19880617: depends on your kafka message encoding type
@gabuglc: my key/message is avro-encoded, because im using kafka-connect with these parameters KEY_CONVERTER: 'io.confluent.connect.avro.AvroConverter' VALUE_CONVERTER: 'io.confluent.connect.avro.AvroConverter'
@gabuglc: Is this the correct value? org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder
@fx19880617: then you can use confluent decoder and set the correct schema registry information. @elon.azoulay might have more inputs
@elon.azoulay: Hi @gabuglc, you can try the following: ``` "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.decoder.prop.schema.registry.rest.url": ""```
@elon.azoulay: substitute the schema registry url for the one you use
@elon.azoulay: this is part of the "streamConfigs"
@elon.azoulay: Does that work for you?
@gabuglc: Aditionally do I need a json_format(message) on transformConfigs?
@shgandhi: @shgandhi has joined the channel

#random

#troubleshooting

#pinot-dev

@mayanks: I am seeing that the integration and unit tests are taking a lot longer and keep failing even after the UI npm fix. Anyone aware of the issue?
@fx19880617:
@mayanks:
@fx19880617: yes
@fx19880617: Unit test fails due to recent github action change, there are some network interface changes
@fx19880617: so dns resolution has some issue
@mayanks: Is that contributing to the long runtime (specially of integ tests) also, or is that separate issue?
@fx19880617: that’s might be separate
@fx19880617: long time integration may come from more integration tests
@jackie.jxt: Long unit test is caused by connection timeout and stuck; long integration test is caused by running more queries with non-empty result:
@mayanks: Are we ok with the runtime though? And is it worth the additional coverage it is providing?
@mayanks: Do we have an increase in coverage due to this @jlli ?
@jlli: Yes, we have included all the types of queries at that time and added more coverage on that
@mayanks: I meant do we have a code cov report that shows the increase in coverage?
@jlli: There is a coverage report in that PR, which shows the coverage increased by 6.74%
@mayanks: Ok. Let's see what the overall test runtime is after we fix the network interface issue. Right now it seems to be taking several hours and still stuck.
@jlli: @mayanks @jackie.jxt I just opened a PR to further reduce the number of queries from 500 to 200:
@mayanks: Thanks @jlli, what's the new runtime, and do we still keep the previous increase in coverage?
@jlli: I removed some of the queries in each query pattern. The code coverage report hasn’t been generated yet. Hopefully it will remain the same.
@mayanks: :+1:
@jlli: We can further reduce the number of queries from 500 to 100, but AFAIK most of the time those integration tests took more time on spinning up the local cluster, especially when the hardware resource is limited
@gandhi16.sharayu: @gandhi16.sharayu has joined the channel
@miliang: @miliang has joined the channel
@miliang: Hey, build on master failed with message ```[INFO] Reactor Summary for Pinot 0.8.0-SNAPSHOT: [INFO] [INFO] Pinot .............................................. SUCCESS [ 15.773 s] [INFO] Pinot Service Provider Interface ................... SUCCESS [ 2.762 s] [INFO] Pinot Segment Service Provider Interface ........... SUCCESS [ 1.093 s] [INFO] Pinot Plugins ...................................... SUCCESS [ 4.414 s] [INFO] Pinot Metrics ...................................... SUCCESS [ 0.145 s] [INFO] Pinot Yammer Metrics ............................... SUCCESS [ 4.240 s] [INFO] Pinot Common ....................................... SUCCESS [ 16.085 s] [INFO] Pinot Input Format ................................. SUCCESS [ 0.993 s] [INFO] Pinot Avro Base .................................... SUCCESS [ 1.109 s] [INFO] Pinot Avro ......................................... SUCCESS [ 0.902 s] [INFO] Pinot Csv .......................................... SUCCESS [ 0.354 s] [INFO] Pinot JSON ......................................... SUCCESS [ 0.344 s] [INFO] Pinot local segment implementations ................ SUCCESS [ 6.542 s] [INFO] Pinot Core ......................................... SUCCESS [ 9.313 s] [INFO] Pinot Server ....................................... SUCCESS [ 4.548 s] [INFO] Pinot Segment Uploader ............................. SUCCESS [ 1.528 s] [INFO] Pinot Segment Uploader Default ..................... SUCCESS [ 15.425 s] [INFO] Pinot Controller ................................... SUCCESS [ 50.838 s] [INFO] Pinot Broker ....................................... SUCCESS [ 4.736 s] [INFO] Pinot Clients ...................................... SUCCESS [ 0.117 s] [INFO] Pinot Java Client .................................. SUCCESS [ 0.474 s] [INFO] Pinot JDBC Client .................................. SUCCESS [ 0.555 s] [INFO] Pinot Batch Ingestion .............................. SUCCESS [ 1.501 s] [INFO] Pinot Batch Ingestion Common ....................... SUCCESS [ 0.393 s] [INFO] Pinot Minion ....................................... SUCCESS [ 1.718 s] [INFO] Pinot Confluent Avro ............................... FAILURE [ 0.586 s] [INFO] Pinot ORC .......................................... SKIPPED [INFO] Pinot Parquet ...................................... SKIPPED [INFO] Pinot Thrift ....................................... SKIPPED [INFO] Pinot Protocol Buffers ............................. SKIPPED [INFO] Pluggable Pinot file system ........................ SKIPPED [INFO] Pinot Azure Data Lake Storage ...................... SKIPPED [INFO] Pinot Hadoop Filesystem ............................ SKIPPED [INFO] Pinot Google Cloud Storage ......................... SKIPPED [INFO] Pinot Amazon S3 .................................... SKIPPED [INFO] Pinot Batch Ingestion for Spark .................... SKIPPED [INFO] Pinot Batch Ingestion for Hadoop ................... SKIPPED [INFO] Pinot Batch Ingestion Standalone ................... SKIPPED [INFO] Pinot Batch Ingestion .............................. SKIPPED [INFO] Pinot Ingestion Common ............................. SKIPPED [INFO] Pinot Hadoop ....................................... SKIPPED [INFO] Pinot Spark ........................................ SKIPPED [INFO] Pinot Stream Ingestion ............................. SKIPPED [INFO] Pinot Kafka Base ................................... SKIPPED [INFO] Pinot Kafka 0.9 .................................... SKIPPED [INFO] Pinot Kafka 2.0 .................................... SKIPPED [INFO] Pinot Minion Tasks ................................. SKIPPED [INFO] Pinot Minion Built-In Tasks ........................ SKIPPED [INFO] Pinot Segment Writer ............................... SKIPPED [INFO] Pinot Segment Writer File Based .................... SKIPPED [INFO] Pinot Tools ........................................ SKIPPED [INFO] Pinot Integration Tests ............................ SKIPPED [INFO] Pinot Perf ......................................... SKIPPED [INFO] Pinot Distribution ................................. SKIPPED [INFO] Pinot Connectors ................................... SKIPPED [INFO] Pinot Spark Connector .............................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:26 min [INFO] Finished at: 2021-04-15T15:49:12-07:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project pinot-confluent-avro: Could not resolve dependencies for project org.apache.pinot:pinot-confluent-avro:jar:0.8.0-SNAPSHOT: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.3.1: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.3.1: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:5.3.1 from/to maven-default-http-blocker (): Blocked mirror for repositories: [confluent (, default, releases+snapshots)] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :pinot-confluent-avro ➜ incubator-pinot git:(master)```
@dlavoie: That’ caused by the latest release of Maven which doesn’t accept `http` repos
@dlavoie: downgrade your maven version and you’ll be fine.
@miliang: Any thoughts?
@jiatao: @jiatao has joined the channel
@shgandhi: @shgandhi has joined the channel
@shgandhi: Hi there, I am facing issues building pinot after recent commits on Master related to build fix. ```[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.1:npm (npm run-script build) on project pinot-controller: Failed to run task: 'npm run-script build' failed. (error code 2) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :pinot-controller``` Can someone guide me?
@dlavoie: Can you scroll up a bit in your build log, you will get the root cause from the `frontend-maven-plugin` execution
@shgandhi: Thanks, it looked liked I had to manually install some node modules. Thanks for the help!
@dlavoie: Can you share more about the error? manually installing modules shouldn’t be a thing
@shgandhi: So, the main issue when I scrolled was ```ERROR in ./node_modules/@material-ui/utils/index.js [INFO] Module not found: Error: Can't resolve './chainPropTypes' in '/Users/shgandhi/GSharayu/incubator-pinot/pinot-controller/src/main/resources/node_modules/@material-ui/utils' [INFO] @ ./node_modules/@material-ui/utils/index.js 74:45-72 [INFO] @ ./node_modules/@material-ui/core/esm/styles/colorManipulator.js [INFO] @ ./node_modules/@material-ui/core/esm/styles/index.js [INFO] @ ./app/theme/index.ts [INFO] @ ./app/App.tsx ```
@shgandhi: and bunch of other node modules
@shgandhi: I tried with npm install
@shgandhi: and got these ```[WARNING] npm WARN [email protected] requires a peer of acorn@^6.0.0 but none is installed. You must install peer dependencies yourself. [WARNING] npm WARN [email protected] requires a peer of eslint-plugin-react-hooks@^2.5.0 || ^1.7.0 but none is installed. You must install peer dependencies yourself. [WARNING] npm WARN [email protected] requires a peer of eslint-plugin-import@^2.22.1 but none is installed. You must install peer dependencies yourself. [WARNING] npm WARN [email protected] requires a peer of [email protected] but none is installed. You must install peer dependencies yourself. [WARNING] npm WARN [email protected] requires a peer of [email protected] || 2.x but none is installed. You must install peer dependencies yourself. [WARNING] npm WARN [email protected] No repository field. [WARNING] npm WARN [email protected] No license field. [ERROR] ```
@fx19880617: fix for current kafka tests:

#announcements

@robeng962: @robeng962 has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]