This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a change to branch master-hadoop3
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from ada0355 KYLIN-4597 Fix NPE when download diagnosis info for a job
new e1f887b Update version from 3.1.0-SNAPSHOT to 3.1.1-SNAPSHOT
new 7d763c1 KYLIN-4611 modify PATTERN_SPARK_APP_URL to Tracking URL,
ignore case
new 713871f KYLIN-4610 update kylin.engine.livy.backtick.quote default
value
new 2f2fe0d KYLIN-4609 setenv.sh add zgc config for big memory
new 9aa83e6 KYLIN-4608 add deletecubefast api for delete 300 cubes fast
new 63513b8 KYLIN-4603 listjobs check cache null
new 5e78722 KYLIN-4275 Fix avg function not working
new 92f0acd KYLIN-4508 Add unit test for core-metrics module & reporters
new 2017cdd KYLIN-4617 Check whether project/jobid exists before download
diagnosis package
new 7ac74a2 KYLIN-4572 Add "build/VERSION" to .gitignore
new fd5fdd0 KYLIN-4551 Provide interfaces to transfer cube/model/project
ownership
new 6691334 KYLIN-4618 Upgrade kylin docker image for kylin 3.1.0
new e3fab3f KYLIN-4581 Add spark and flink engine test case for release
test
new 2c3c0ac KYLIN-4616 Auto detect the default value of config
kylin.source.hive.databasedir
new 250ad90 KYLIN-4342 Fix incorrect database
new debb8d2 KYLIN-4619 Make shrunken dict able to coexist with mr-hive
global dict
new 19d5d53 KYLIN-4527 Beautify the drop-down list of the cube on query
page
new a03462b minor,update pre-defined name/pwd comment
new 7621e91 KYLIN-4627 add docker document in README.md
new 4948ebf KYLIN-4633 Fix CI
new a741991 KYLIN-4606 throw olap exception msg and pushdown exception
msg when olap query and pushdown both error
new 925ac86 KYLIN-4612 support job status writing to kafka
new f66c0a2 KYLIN-4526 Enhance get the hive table rows
new 4cd3e1a KYLIN-4635 set org.apache.kylin default log level be info
new 39d1596 KYLIN-4576 add some hint of password length
new 58bee00 Create maven.yml
new 52400e3 KYLIN-4653 Make the capacity for the LinkedBlockingQueue of
BlockingReservoir configurable
new 9af33f2 KYLIN-4656 Guava classpath conflict caused by kylin-jdbc
3.1.0 jar
new 3861f7b KYLIN-4653 Make the capacity for the LinkedBlockingQueue of
BlockingReservoir configurable
new 1169035 Fix travis ci job log exceeded maximum log length
new 5ac99f6 minor, api format
new 625477d KYLIN-4684 Fix web ui bug when create streamig table
new 6309e1b KYLIN-4678 continue execute job when clean job sub step has
error
new ad7daff KYLIN-4677 storagecleanupjob throw NPE
new c8bda9c KYLIN-4657 dead-loop in
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork
new c340c61 set flink job name shown in resource manager
new b98b91a KYLIN-4634 Fail to specify cube in model of low priority to
query
new 66bbbb2 修改PatternedLogger构造方法，避免spark_job_id与flink_job_id互相覆盖
new 6feedda 修正命中pattern
new 244bfed Revert "minor, api format"
new 110efa0 minor, api format
new 99e1346 KYLIN-4686 clean metadata support to delete all jobs
new bba89ea KYLIN-4687 add unify clean sh to excute some clean shells
new 5940d93 KYLIN-4685 return user friendly msg when stackoverflowerror
new 297e55d KYLIN-4687 addendum - add apache license header
new cfed9ba KYLIN-4700 Wrong engine type for realtime streaming
new 50cc660 Fix sonar vulunerability issues (#1363)
new 1a200ad minor, fix sonar reported bugs
new d09bb77 minor, fix a sonar reported bug
new cf34946 KYLIN-4515 fix error with sending mail on ssl (#1214)
new f7287b3 KYLIN-4549 Show column cardinality in rowkeys area of
advanced settings
new 78e6edc KYLIN-4585 Add cube count column for project table (#1277)
new 362d4df KYLIN-4550 Provide advanced refresh interface inside the
refresh panel
new e782ea8 minor, fix sonar reported bugs
new 994883e KYLIN-4671 update log level for ignore too many logs
new cc59359 KYLIN-4628: Fail to use custom measure type when specifying
cube to query
new 6a8c371 KYLIN-4557 Refactor JobService to improve code readability
new d84d025 KYLIN-4558 get all chained executable jobs through job API
new 86ed770 KYLIN-4559 show cardinality and lookup snapshot job on job
new 0ae1a98 KYLIN-4560 support to re-run/delete cardinality and lookup
snapshot job
new d80fddb KYLIN-4561 overall job number statistics in monitor page is
incorrect after change the job status
new 95c7b7c KYLIN-4731 Kylin query failing with 'NPE while executing SQL'
new b1c98ae KYLIN-4716 Optimize the project page
new 7e54262 Update the documentation to specify a given container name
for docker run/exec commands
new ef55bbb Update AbstractHadoopJob.java
new 0063f0a KYLIN-4753 Merging job stop working after Kylin upgrade
new 781df39 KYLIN-4756 user/group page has duplicate information between
adjacent page numbers
new 3975534 KYLIN-4679 clean hive table support hive table prefix
new e5b17be KYLIN-4755 Error while compiling generated Java code when
using Kylin UDF in "case when"
new 5eb69d8 KYLIN-4714 Failed to revoke role access of the project
new f4fbe98 KYLIN-4683 Fail to consume kafka when partition number get
larger
new 65b6016 KYLIN-4712 Optimize Optimize CubeMetaIngester.java CLI
new 57f53b0 KYLIN-4648 Upgrade Spark to latest 2.3 or 2.4 version
new ec98071 KYLIN-4757 Fix decimal precision
new da9ab94 KYLIN-4689 Purge inconsistent lock before check
segmentBuildJobCheckList
new 031643f KYLIN-4551 addendum - fix WriteConflictException thrown by
update model owner
new 126aae2 KYLIN-4712 Fix some bugs when ingest cube
The 77 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../workflows/maven.yml | 22 +-
.gitignore | 2 +
.travis.yml | 4 +-
README.md | 29 +-
assembly/pom.xml | 2 +-
...t-availability.sh => clean-kylin-dirty-data.sh} | 24 +-
build/bin/download-spark.sh | 4 +-
build/bin/find-hive-dependency.sh | 2 +
build/bin/kylin.sh | 1 +
build/conf/kylin-server-log4j.properties | 4 +-
build/conf/kylin-spark-log4j.properties | 2 +-
build/conf/kylin-tools-log4j.properties | 2 +-
build/conf/setenv.sh | 3 +
build/smoke-test/smoke-test.sh | 1 +
build/smoke-test/testBuildCube.py | 51 +-
build/smoke-test/testQuery.py | 56 +-
cache/pom.xml | 2 +-
core-common/pom.xml | 7 +-
.../java/org/apache/kylin/common/KylinConfig.java | 20 +-
.../org/apache/kylin/common/KylinConfigBase.java | 34 +-
.../kylin/common/kafka/KafkaMsgProducer.java | 92 +++
.../apache/kylin/common/livy/LivyRestBuilder.java | 8 +-
.../apache/kylin/common/livy/LivyRestExecutor.java | 8 +-
.../common/metrics/perflog/PerfLoggerFactory.java | 2 +-
.../common/persistence/AutoDeleteDirectory.java | 27 +-
.../common/persistence/JDBCResourceStore.java | 12 +-
.../common/persistence/ResourceParallelCopier.java | 14 +-
.../kylin/common/persistence/ResourceStore.java | 19 +-
.../kylin/common/persistence/ResourceTool.java | 2 +-
.../apache/kylin/common/restclient/RestClient.java | 2 +-
.../java/org/apache/kylin/common/util/Array.java | 6 +-
.../java/org/apache/kylin/common/util/Bytes.java | 2 +-
.../kylin/common/util/CliCommandExecutor.java | 2 +-
.../org/apache/kylin/common/util/Dictionary.java | 2 +-
.../apache/kylin/common/util/HiveCmdBuilder.java | 5 +
.../org/apache/kylin/common/util/MailService.java | 1 +
.../java/org/apache/kylin/common/util/Primes.java | 2 +-
.../org/apache/kylin/common/util/ServerMode.java | 2 +-
core-cube/pom.xml | 2 +-
.../java/org/apache/kylin/cube/CubeManager.java | 10 +
.../java/org/apache/kylin/cube/CubeSegment.java | 2 +-
.../cube/inmemcubing/AbstractInMemCubeBuilder.java | 8 +-
.../kylin/cube/inmemcubing/CuboidResult.java | 13 +-
.../cube/inmemcubing/ICuboidGTTableWriter.java | 8 +-
.../kylin/cube/inmemcubing/InMemCubeBuilder.java | 25 +-
.../kylin/cube/inmemcubing2/InMemCubeBuilder2.java | 14 +-
.../java/org/apache/kylin/cube/model/CubeDesc.java | 40 +-
.../gridtable/GTAggregateTransformScanner.java | 41 +-
.../gridtable/GTTwoLayerAggregateScanner.java | 2 +
.../benchmark/SortedGTRecordGenerator.java | 13 +-
.../kylin/cube/inmemcubing/MemDiskStoreTest.java | 11 +-
.../kylin/gridtable/SimpleGridTableTest.java | 84 ++-
core-dictionary/pom.xml | 2 +-
.../org/apache/kylin/dict/DictionaryGenerator.java | 1 +
.../apache/kylin/dict/GlobalDictionaryBuilder.java | 11 +-
.../org/apache/kylin/dict/lookup/LookupTable.java | 2 +-
.../dict/lookup/cache/RocksDBLookupTable.java | 2 +-
core-job/pom.xml | 2 +-
.../apache/kylin/job/common/PatternedLogger.java | 12 +-
.../kylin/job/constant/ExecutableConstants.java | 3 +
.../kylin/job/execution/AbstractExecutable.java | 9 +
.../kylin/job/execution/CheckpointExecutable.java | 9 -
.../kylin/job/execution/ExecutableManager.java | 58 ++
.../job/impl/threadpool/DefaultFetcherRunner.java | 4 +-
core-metadata/pom.xml | 2 +-
.../java/org/apache/kylin/measure/MeasureType.java | 2 +-
.../kylin/measure/basic/BasicMeasureType.java | 2 +-
.../kylin/measure/bitmap/BitmapMeasureType.java | 2 +-
.../measure/bitmap/RetentionPartialResult.java | 2 +-
.../measure/dim/DimCountDistinctMeasureType.java | 1 +
.../apache/kylin/measure/hllc/HLLCMeasureType.java | 2 +-
.../measure/map/bitmap/BitmapMapMeasureType.java | 2 +-
.../apache/kylin/measure/raw/RawMeasureType.java | 3 +-
.../apache/kylin/measure/topn/TopNMeasureType.java | 34 +-
.../kylin/metadata/cachesync/Broadcaster.java | 12 +-
.../kylin/metadata/model/DataModelManager.java | 12 +-
.../kylin/metadata/project/ProjectManager.java | 12 +
.../kylin/metadata/streaming/StreamingConfig.java | 2 +-
.../kylin/metadata/cachesync/BroadcasterTest.java | 6 +-
core-metrics/pom.xml | 2 +-
.../kylin/metrics/lib/impl/BlockingReservoir.java | 10 +-
.../kylin/metrics/lib/impl/TimedRecordEvent.java | 32 +
core-storage/pom.xml | 2 +-
.../kylin/storage/gtrecord/CubeTupleConverter.java | 8 +-
.../storage/translate/DerivedFilterTranslator.java | 10 +-
.../kylin/storage/gtrecord/DictGridTableTest.java | 26 +-
cube-migration/pom.xml | 2 +-
datasource-sdk/pom.xml | 2 +-
.../sdk/datasource/adaptor/DefaultAdaptor.java | 18 +-
docker/Dockerfile | 9 +-
docker/Dockerfile_hadoop | 3 +-
docker/README.md | 24 +-
docker/build_image.sh | 3 +-
docker/entrypoint.sh | 7 +-
docker/run_container.sh | 3 +-
engine-flink/pom.xml | 2 +-
.../engine/flink/FlinkBatchCubingJobBuilder2.java | 2 +-
.../apache/kylin/engine/flink/FlinkExecutable.java | 7 +-
engine-mr/pom.xml | 2 +-
.../java/org/apache/kylin/engine/mr/CubingJob.java | 18 +-
.../apache/kylin/engine/mr/JobBuilderSupport.java | 6 +-
.../kylin/engine/mr/LookupSnapshotBuildJob.java | 9 -
.../kylin/engine/mr/common/AbstractHadoopJob.java | 4 -
.../kylin/engine/mr/common/CubeStatsReader.java | 19 +-
.../engine/mr/common/DictionaryGetterUtil.java | 2 +-
.../kylin/engine/mr/common/JobInfoConverter.java | 20 +-
.../engine/mr/common/MapReduceExecutable.java | 25 +-
.../mr/steps/BuildGlobalHiveDictPartBuildJob.java | 2 +-
.../steps/ExtractDictionaryFromGlobalMapper.java | 2 +-
.../mr/streaming/ColumnarSplitDataReader.java | 2 +-
.../kylin/engine/mr/streaming/DictsReader.java | 44 +-
.../kylin/engine/mr/streaming/RowRecordReader.java | 43 +-
engine-spark/pom.xml | 2 +-
.../apache/kylin/engine/spark/SparkExecutable.java | 3 +-
.../kylin/engine/spark/SparkExecutableLivy.java | 3 +-
examples/test_case_data/sandbox/kylin.properties | 2 +-
external/pom.xml | 2 +-
external/shaded-guava/pom.xml | 6 +-
jdbc/pom.xml | 3 +-
.../config/production/kylin-more/kylinSecurity.xml | 3 +-
.../example/config/kylin-more/kylinSecurity.xml | 3 +-
kylin-it/pom.xml | 5 +-
.../kylin/realtime/BuildCubeWithStreamV2.java | 2 +-
.../apache/kylin/restclient/ITRestClientTest.java | 21 +
metrics-reporter-hive/pom.xml | 27 +-
.../kylin/metrics/lib/impl/hive/HiveProducer.java | 7 +-
.../metrics/lib/impl/hive/HiveProducerRecord.java | 141 ++---
.../lib/impl/hive/HiveReservoirReporter.java | 39 +-
.../tool/metrics/systemcube/HiveTableCreator.java | 3 +-
.../lib/impl/hive/HiveProducerRecordTest.java | 81 +++
.../metrics/lib/impl/hive/HiveProducerTest.java | 161 ++++++
.../lib/impl/hive/HiveReservoirReporterTest.java | 88 +++
metrics-reporter-kafka/pom.xml | 21 +-
.../impl/kafka/KafkaActiveReserviorListener.java | 8 +
.../lib/impl/kafka/KafkaReservoirReporter.java | 6 +-
.../lib/impl/kafka/KafkaReservoirReporterTest.java | 79 +++
pom.xml | 4 +-
query/pom.xml | 2 +-
.../kylin/query/relnode/OLAPAggregateRel.java | 3 +
.../apache/kylin/query/relnode/OLAPContext.java | 12 +-
.../apache/kylin/query/routing/QueryRouter.java | 4 +-
.../kylin/query/routing/RealizationChooser.java | 9 +
.../java/org/apache/kylin/query/udf/ConcatUDF.java | 2 +-
.../java/org/apache/kylin/query/udf/MassInUDF.java | 2 +-
.../org/apache/kylin/query/udf/VersionUDF.java | 2 +-
.../org/apache/kylin/query/util/QueryUtil.java | 6 +
server-base/pom.xml | 2 +-
.../kylin/rest/controller/CubeController.java | 80 ++-
.../kylin/rest/controller/DiagnosisController.java | 5 +-
.../kylin/rest/controller/ModelController.java | 56 ++
.../kylin/rest/controller/ProjectController.java | 57 ++
.../apache/kylin/rest/job/MetadataCleanupJob.java | 4 +-
.../apache/kylin/rest/job/StorageCleanupJob.java | 37 +-
.../java/org/apache/kylin/rest/msg/CnMessage.java | 8 +
.../java/org/apache/kylin/rest/msg/Message.java | 8 +
.../org/apache/kylin/rest/request/SQLRequest.java | 2 +-
.../org/apache/kylin/rest/service/AclService.java | 4 +-
.../org/apache/kylin/rest/service/CubeService.java | 24 +-
.../kylin/rest/service/DiagnosisService.java | 25 +-
.../org/apache/kylin/rest/service/JobService.java | 642 +++++++--------------
.../apache/kylin/rest/service/ProjectService.java | 13 +
.../apache/kylin/rest/service/QueryService.java | 12 +-
.../apache/kylin/rest/service/TableService.java | 16 +-
.../org/apache/kylin/rest/util/AclEvaluate.java | 25 +-
.../kylin/rest/job/MetadataCleanupJobTest.java | 4 +-
server/pom.xml | 2 +-
server/src/main/resources/kylinMetrics.xml | 8 +-
server/src/main/resources/kylinSecurity.xml | 4 +-
.../rest/controller/AccessControllerTest.java | 106 +++-
.../kylin/rest/controller/CubeControllerTest.java | 25 +-
.../rest/controller/ProjectControllerTest.java | 42 ++
.../kylin/rest/service/AccessServiceTest.java | 53 ++
source-hive/pom.xml | 2 +-
.../kylin/source/hive/BeelineHiveClient.java | 39 +-
.../apache/kylin/source/hive/CLIHiveClient.java | 19 +-
.../apache/kylin/source/hive/HiveInputBase.java | 6 +-
.../apache/kylin/source/hive/MRHiveDictUtil.java | 4 +-
source-jdbc/pom.xml | 2 +-
.../kylin/source/jdbc/extensible/JdbcExplorer.java | 12 +-
source-kafka/pom.xml | 2 +-
storage-hbase/pom.xml | 2 +-
.../kylin/storage/hbase/HBaseResourceStore.java | 12 +-
.../storage/hbase/lookup/HBaseLookupTable.java | 2 +-
storage-stream/pom.xml | 2 +-
stream-coordinator/pom.xml | 2 +-
.../coordinator/coordinate/BuildJobSubmitter.java | 57 ++
stream-core/pom.xml | 2 +-
.../columnar/ColumnarMemoryStorePersister.java | 12 +-
.../storage/columnar/ColumnarStoreDimDesc.java | 13 +-
.../storage/columnar/ColumnarStoreMetricsDesc.java | 13 +-
.../columnar/FSInputGeneralColumnDataReader.java | 6 +-
.../storage/columnar/FragmentCuboidReader.java | 12 +-
.../core/storage/columnar/SegmentMemoryStore.java | 29 +-
.../compress/FSInputLZ4CompressedColumnReader.java | 13 +-
.../compress/FSInputNoCompressedColumnReader.java | 18 +-
.../compress/FSInputRLECompressedColumnReader.java | 12 +-
.../compress/LZ4CompressedColumnReader.java | 8 +-
.../compress/NoCompressedColumnReader.java | 4 +
.../compress/RunLengthCompressedColumnReader.java | 5 +
.../kylin/stream/core/util/RecordsSerializer.java | 1 +
.../columnar/compress/LZ4CompressColumnTest.java | 4 +-
.../columnar/compress/NoCompressColumnTest.java | 5 +-
.../compress/RunLengthCompressColumnTest.java | 4 +-
stream-receiver/pom.xml | 2 +-
stream-source-kafka/pom.xml | 2 +-
.../source/kafka/consumer/KafkaConnector.java | 10 +-
tomcat-ext/pom.xml | 2 +-
tool-assembly/pom.xml | 2 +-
tool/pom.xml | 2 +-
.../org/apache/kylin/tool/CubeMetaIngester.java | 33 +-
.../org/apache/kylin/tool/MetadataCleanupJob.java | 9 +-
.../kylin/tool/extractor/MrJobInfoExtractor.java | 29 +-
.../kylin/tool/query/ProbabilityGenerator.java | 4 +
webapp/app/js/controllers/cube.js | 18 +
webapp/app/js/controllers/cubeAdvanceSetting.js | 75 ++-
webapp/app/js/controllers/cubes.js | 46 ++
webapp/app/js/controllers/job.js | 50 +-
webapp/app/js/controllers/projects.js | 23 +-
webapp/app/js/controllers/query.js | 15 +-
webapp/app/js/controllers/sourceMeta.js | 6 +-
webapp/app/js/controllers/userGroup.js | 91 ++-
webapp/app/js/model/jobConfig.js | 2 +
webapp/app/js/model/jobListModel.js | 9 +-
webapp/app/js/model/projectConfig.js | 1 +
webapp/app/js/services/cubes.js | 1 +
webapp/app/js/services/projects.js | 3 +-
webapp/app/less/app.less | 17 +-
webapp/app/partials/admin/change_pwd.html | 8 +-
webapp/app/partials/admin/user_create.html | 2 +-
.../partials/cubeDesigner/advanced_settings.html | 9 +
webapp/app/partials/cubes/cube_detail.html | 19 +-
webapp/app/partials/jobs/job_refresh.html | 182 ++++--
webapp/app/partials/projects/project_detail.html | 21 +
webapp/app/partials/projects/projects.html | 5 +-
webapp/app/partials/query/query.html | 13 +-
webapp/app/partials/tables/table_detail.html | 2 +-
236 files changed, 3027 insertions(+), 1370 deletions(-)
copy build/conf/kylin.properties => .github/workflows/maven.yml (59%)
copy build/bin/{check-port-availability.sh => clean-kylin-dirty-data.sh} (53%)
create mode 100644
core-common/src/main/java/org/apache/kylin/common/kafka/KafkaMsgProducer.java
rename {tool =>
metrics-reporter-hive}/src/main/java/org/apache/kylin/tool/metrics/systemcube/HiveTableCreator.java
(99%)
create mode 100644
metrics-reporter-hive/src/test/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducerRecordTest.java
create mode 100644
metrics-reporter-hive/src/test/java/org/apache/kylin/metrics/lib/impl/hive/HiveProducerTest.java
create mode 100644
metrics-reporter-hive/src/test/java/org/apache/kylin/metrics/lib/impl/hive/HiveReservoirReporterTest.java
create mode 100644
metrics-reporter-kafka/src/test/java/org/apache/kylin/metrics/lib/impl/kafka/KafkaReservoirReporterTest.java
