This is an automated email from the ASF dual-hosted git repository. shaofengshi pushed a change to branch master-hadoop3.1 in repository https://gitbox.apache.org/repos/asf/kylin.git.
discard 978bb37 KYLIN-2565, upgrade to Hadoop3.1 add ec8186e KYLIN-3300 Upgrade jackson-databind add 3282faf KYLIN-3301 Upgrade opensaml add 0029d8a KYLIN-3278 Revert "KYLIN-3084, fix issue by using distribute by" add 6cc59cf KYLIN-3304 temporarily disable CacheServiceTest add 236e6f5 KYLIN-3306: Fix the rarely happened unit test exception of generic algorithm add 24042e2 KYLIN-3270: add integration test for optimize job add cfb21d7 KYLIN-3292 The update button should be disabled with none key and value in setting config dialog (#116) add a7a0baa KYLIN-3288 specify mapreduce.job.queuename when submit sqoop job add 03316e2 KYLIN-3314 refactor code for cube planner algorithm add 8350de4 KYLIN-3293, fix FixedLenHexDimEnc that return a wrong code length leads to cut bytes error. add 565a442 KYLIN-3320 fix CubeStatsReader cannot print stats properly for some cube add aff6f42 minor, add main func for EncryptUtil add 2e49128 minor, fix query handling right outer join case add a6bfc9d KYLIN-3311 refactor Resource.checkAndPutResourceImpl() to throw WriteConflictException add febabc4 minor, add null case. add cfe84a1 KYLIN-3277 Kylin should override hiveconf settings when connecting to hive using jdbc add 417c4b6 KYLIN-3277 Code review add 343d63d KYLIN-3324 fix NegativeArraySizeException in CreateDictionaryJob$2.getDictionary() add 37d3cd9 minor, add test util SetAndUnsetSystemProp add 709e564 KYLIN-3315 allow each project to set its own source add 5552164 KYLIN-3315 allow each project to set its own source add be3588b KYLIN-3296 Fix the split rowkey problem add 1e884f2 Revert "KYLIN-3296 Fix the split rowkey problem" add 2cda6c3 KYLIN-3327 Upgrade surefire version to 2.21.0 add 7540cbd KYLIN-3339, decoupling SQL keywords set with specify implementation. add 5ab78cb KYLIN-3343 support RDBMS on GUI (#131) add 6fb75b6 KYLIN-3331 Add script check-hive-usability.sh to avoid script hangs add 9509989 KYLIN-3328 Upgrade the metadata of sample cube to latest add 9d88e47 minor, fix exception swallow in CubeDesc.init() add 71397fd KYLIN-3289 Refactor the storage garbage clean up code add a9dcad1 KYLIN-3337 Use an auto-closeable style API to replace KylinConfig.setKylinConfigThreadLocal() add 143cfdc Minor, add config for enable and disable auto merge add b1c8944 KYLIN-3340, enhance hql check add 7e8d5a5 Minor, remove TarGZUtil.java add b9bcc92 CubeHFileJob should use currentHBaseConfiguration but not new create hbase configuration add 7217d7f KYLIN-3294 Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes add 2ca5b98 KYLIN-3369 reduce data size from kafka add f2382ea KYLIN-3369 code refine add 5611760 KYLIN-3368 Move Spark cubing metadata dump to job folder add 026db17 KYLIN-3367 add the compatibility for new version of hbase add 877cab0 minor, make the metrics parameter name consistent add e20e244 KYLIN-3345 Use Apache Parent POM 19 add faf7064 KYLIN-3363 fix wrong partition condition appended in JDBC Source add 3681089 KYLIN-3380 JDBC source - add a configuration file for sqoop add 6820f8d KYLIN-3315 fix table init add bbb71a8 KYLIN-3348 fix 'missing LastBuildJobID' error add 1a1c8a9 KYLIN-3352 better filter transform for better seg pruning add c027831 KYLIN-3354 escape double-quoted defaultCatalog keyword add 0efcdca KYLIN-3315 fix table init to avoid deadlock on project manager add e08c106 Minor, fix wrong method name of cube's input record size add 398401f Minor, add member "displayCubeName" for job instance add 2b1e9f4 Minor, make query response with "Unknown error" if exception without message occurs add 999f1f0 KYLIN-3250 Upgrade jetty version to 9.3.22.v20171030 add 4c67905 KYLIN-3372 upgrade jackson databind version to 2.9.5 add d1c68f6 KYLIN-3336 add 8325ae9 KYLIN-3295, clean unused class:SQLDigestUtil add f0f588f KYLIN-3373 Some improvements for lookup table - UI part change add 179eddb KYLIN-3374 Some improvements for lookup table - metadata change add 777f8fa KYLIN-3375 Some improvements for lookup table - build change add 5a96f8b KYLIN-3376 Some improvements for lookup table - query change add b7d2cb7 KYLIN-3377 Some improvements for lookup table - snapshot management add 71d416a KYLIN-3373 Some improvements for lookup table - UI part change - Bug fix for add snapshot desc add 20c66d8 KYLIN-3375 Some improvements for lookup table - build change - Fix issue when latest ext snapshot exist add 18316ec KYLIN-3375 Some improvements for lookup table - build change - Fix rocksdb cache UT issue add c7d5ea2 Analytic table name ArrayIndexOutOfBoundsException add e0f29e2 KYLIN-3137 Spark cubing refine add 8205fac KYLIN-3137 update kylin.properties for spark 2.1 add e3dcafb KYLIN-3161 Enforce global dictionary for bitmap count distinct column(as data type is not int) add 2a8d994 KYLIN-3396 NPE throws when materialize lookup table to HBase add 0fdf55a KYLIN-3137 Spark cubing without hive add e5accc5 minor,update query chart config add f85d561 KYLIN-3378 Support Kafka table join with Hive tables add 995a719 KYLIN-3137 allowing config Spark storage level add 372d1bb KYLIN-3401 change zip compress tool from java.util.zip to Apache commons-compress library add 9fa82a4 minor, fix NoClassDefFoundError: org/apache/commons/compress/archivers/examples/Archiver add 93247fa KYLIN-3094 Upgrade zookeeper version from 3.4.8 to 3.4.12 add 38b47c7 KYLIN-3093 Upgrade curator to 2.12 add ff122eb KYLIN-3398 Fix inaccurate arithmetic operations in LookupTableToHFileJob#calculateShardNum add de17b6e KYLIN-3093 add curator-client add f6b1dfb KYLIN-3115 Incompatible RowKeySplitter initialize between build and merge job add 2b531d9 minor,update advance snapshot table title add fd7cc17 KYLIN-1948, add test to cover the case. add c8de73d KYLIN-3388 remove create redistribute flat hive table step if not set distribute column add ec927e3 KYLIN-3364 make it consistent with hive for BigDecimalSumAggregator dealing with null add e3388f9 KYLIN-3359 Support sum(expression) if possible add f0c7e0f KYLIN-3359 add unit test & integration test add 1c9b679 KYLIN-3360 correct count(column) add 189f7ae KYLIN-3360 add integration test add d3adc52 KYLIN-3362 support dynamic dimension push down add 0d4d1ef KYLIN-3362 add integration test add ed1882f KYLIN-3358 add a trigger kylin.query.enable-dynamic-column with default value false for coprocessor backward compatibility add e47f2cf APACHE-KYLIN-3359 minor fix add a9557a6 APACHE-KYLIN-3359 change tupleExpressionList to tupleExpressionMap & keep dynamicCols and tupleExpressionMap independent in GTScanRequest add 4c22115 KYLIN-3408 update LICENSE add 52bceb8 Revert "KYLIN-3161 Enforce global dictionary for bitmap count distinct column(as data type is not int)" add 6d8892d KYLIN-3404 The last optimized time not showing up on cube planner tab add 1da7e28 KYLIN-3400 wipeCache and createCubeDesc make deadlock add 44ac227 KYLIN-3321, Set env MALLOC_ARENA_MAX=4 by default. add fbd36de KYLIN-3347 : calcite function QueryService Exception add 01df768 general PTN_HAVING_FUNCTION removed and then add CURRENT_TIMESTAMP in PTN_HAVING_ESCAPE_FUNCTION add 0f6273e KYLIN-3415 Remove external module add d680a4a KYLIN-3399, makes the code that added in KYLIN-2474 more intuitive. add 40a5935 changed unsafe code to safety add ef28440 satisfy multi-path of domain for kylin connection add 4f2992d fix bug :: domain contains projectName add dbaf37e Minor, fix compile error add a602afd KYLIN-3358 make the config kylin.query.enable-dynamic-column effective at project level add eddab37 KYLIN-3421 improve the fetcher runner in job scheduler add c5fd42a KYLIN-3421 fix IT add cedbdc5 KYLIN-3408 Bump Kylin version to 2.4.1 add 5a089ad KYLIN-3408 Bump Kylin version to 2.5.0 add a0595c0 KYLIN-3425 update Kaisen Kang public keys to KEYS add 959ba2d KYLIN-3025 add 2c08e7f KYLIN-3384 Allow setting REPLICATION_SCOPE on newly created tables add 7c1d902 update docs url add 40dd2be KYLIN-3440 Couldn't merge if there is an empty segment (#161) add 54cd2c7 KYLIN-3427 covnert to HFile in Spark add bec6e59 KYLIN-3427 fix UT in AdminServiceTest add 59354ac KYLIN-3428 too large dict file cause OOM add 1845ee0 KYLIN-3391 BadQueryDetector only detect first query add fdbe369 KYLIN-3390 QueryInterceptorUtil.queryInterceptors is not thread safe add f487d80 KYLIN-2662 fix NegativeArraySizeException in TrieDictionaryForest add 05baf21 KYLIN-3370 enhance segment pruning add d1ed107 KYLIN-3423 Performance improvement in FactDistinctColumnsMapper add 31073af KYLIN-3403 Use IntegerCodeSystem for date type filter add 01c3771 minor, update TOP_N measure default encoding add ee7f2f4 KYLIN-3382 Yarn job link wasn't displayed when job is running add 22f2d6e KYLIN-3382 code refine add a565fda KYLIN-3427 Bug fix for covnerting to HFile in Spark add 4bda69e KYLIN-3441 Merge cube segments in Spark add d5affd9 KYLIN-3438 mapreduce.job.queuename does not work at 'Convert Cuboid Data to HFile' Step add 8a9a333 KYLIN-3255 Cannot save cube add 85c3895 KYLIN-3255 cannot save cube and unit test add 4b5fe6d KYLIN-3186 add partition_time other feature, abstract the 'initialPartitionSetting' method and enable the partition format editable of all data type add 7f2aa5a KYLIN-3186 expand partition-date-column format add 27eb8cd KYLIN-2933 fix compilation against the Kafka 1.0.0 release add 95d2a5b KYLIN-3457 Distribute by multi column if not set distribute column add 11642df KYLIN-3457 use cube level config (#170) add c9d9b2e KYLIN-3441 Merge segments in spark add f27a92b KYLIN-3259 when delete cube, remove it from hybrid add 036e26c KYLIN-3414 Optimize the cleanup of project L2 cache add 9bc8d12 KYLIN-2998 Kill spark app when job discarded add 312eaae KYLIN-3452 Optimize spark cubing memory footprint add b1032eb KYLIN-2998 bug fix add 88a242f KYLIN-3418 Adjust hybrid API add f5e28fb KYLIN-3418 User interface for hybrid model - Frontend add 1cd0026 KYLIN-3453 Improve cube size estimation for topn,count distinct add 03f7f67 KYLIN-3464 Less user confirmation add 25b300c Merge pull request #179 from Emiya0306/master add de693a7 KYLIN-3478 Enhance backwards compatibility. add 44926b7 KYLIN-3472 Optimize TopN merge in for Spark engine add 2b9aedc KYLIN-3461 metastore.sh refresh-cube-signature not updating cube signature as expected add bccfb7f KYLIN-3475 Make calcite connection is more configurable. add 9fec0c4 KYLIN-3355 Improve the HTTP return code add 23ff19d KYLIN-3385 fix sum1 error add 95f35fc KYLIN-3379 enhance the test coverage of timestamp add. add 122b4d0 KYLIN-3479 Model can save when Kafka partition date column not select add 4d61c30 KYLIN-3483, fix the loss of precision in HLLCSerializer. add b938a1c KYLIN-3476 fix TupleExpression verification when parsing sql add e7bacd3 KYLIN-3463 use LazyOutputFormat to prevent to create zero-sized default output for optimize jobs add 050f1c1 KYLIN-3485 Make unloading table more flexible add 9a92f49 KYLIN-3471 Merge dictionary using Mapreduce add 74ac1cd KYLIN-3471 Merge dictionary using Spark add 6adf1b3 KYLIN-3481 Shaded dependencies should be optional add 7e0cbad Minor, fix rat check failed add 3fe29d0 KYLIN-3471 Minor, remove duplicate code add ddfed27 KYLIN-3492, fix typo in KylinConfigBase.getDefaultVarcharPrecision. add 367fdfb KYLIN-3434 Support prepare statement in Kylin server side add 0b5d255 KYLIN-3470 Add cache for execute and execute_output to speed up list job api add 6483798 KYLIN-3317, replace UUID.randomUUID with new UUID(ThreadLocalRandom.current().nextLong(), ThreadLocalRandom.current().nextLong()). add d2d2ac7 Update README badges add b4f2155 Enable Travis CI cache to speed up CI checks add 2eba8e2 KYLIN-3500 fix dumplicated tablename at createSqoopToFlatHiveStep when use jdbc datasource add 55d8ada KYLIN-3424 invoke addCubingGarbageCollectionSteps in the cleanup step for HBaseMROutput2Transition add 76c9c96 KYLIN-3446 Connect to HBase out of Spark Signed-off-by: shaofengshi <shaofeng...@apache.org> add c9337cb KYLIN-3489 improve the efficiency of enumerating dictionary values by pre-order visiting add c9d7f5e KYLIN-3490 introduce DictionaryEnumerator to answer single encoded column related queries which will not hit cuboid add 2372b96 KYLIN-3505 Fix wrong usage of cache in DataType add 64dfa7b KYLIN-3507 Avoid NPE when project is not found add 024a60d KYLIN-3449 Allow deleting a NEW status orphan segment add e8e2052 KYLIN-3071: Add config to reuse dict to reduce dict size add a334ee4 KYLIN-3442 Fact distinct columns in Spark add 637f45d KYLIN-3491 add a shrunken global dictionary step to improve the encoding process add 80c60ee KYLIN-3491 enable shrunken gd in ci test add 8ecaba5 KYLIN-3454 #5525 fix potential thread-safe problem add 32c6593 KYLIN-3510 Correct sqoopHome at 'createSqoopToFlatHiveStep' add c5424ce KYLIN-3477 Save spark job counter to hdfs add 5a2b76c KYLIN-3516 Fix job status not updated after job discarded add d26d321 KYLIN-3503 fix missing java.util.logging.config.file when starting kylin instance add 88db74a KYLIN-3480 change the sql conformance of calcite from default to lenient to enable groupby & orderby ordinal add d5aebf3 KYLIN-3456 cube level's snapshot config does not work add 877cb8a KYLIN-3458 fix incomplete log info problem that is caused by kylin.job.retry add 2f6eff6 KYLIN-3436 code refactor add 3de7c4f KYLIN-3509 Allocate more memory for merge-dictionary step add 2889e36 KYLIN-3451 the cloned cube don't have Mandatory Cuboids add c897277 KYLIN-3482 Unclosed SetAndUnsetThreadLocalConfig in Spark engine add f1d2ee4 minor, move DictColDeduperTest.java to the right folder add 91db31c KYLIN-3187 Forbid calls to JDK APIs that use the default locale, time zone or character set add a0fdac4 minor, fix grunt script - font build add 119053b Merge pull request #218 from Emiya0306/master add 6d2738e KYLIN-3522 PrepareStatement cache issue add 8a266f2 KYLIN-3488 Support Mysql as Kylin metadata storage add 101758d KYLIN-3488 code review add 94afb6b KYLIN-3521 Enable Cube Planner by default add e7e8e62 KYLIN-3535 replace and uncomment the web server port add fff673f KYLIN-3536 PrepareStatement cache issue when there are new segments built add 6d00ed5 KYLIN-3521 fix ut add 85801fa KYLIN-3504 update Yanghong Zhong public keys to KEYS add 5fb5b0c KYLIN-3488 bug fix for using MessageFormat.forma add 7a77141 KYLIN-3258 No check for duplicate cube name when creating a hybrid cube add 37a4966 KYLIN-3529 Show more error message add d164225 KYLIN-3534 Don't compress fact distinct output file add 5faddf6 KYLIN-3527 Hybrid allows 0,1 cube add bffaa8a KYLIN-3539 Hybrid segment overlap check new 616bb38 KYLIN-2565, upgrade to Hadoop3.0 new a094850 KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT new 7eeed17 Update scripts for HDP 3.0 new 00e8bd7 KYLIN-3518 Fix Coprocessor NPE problem on hbase 2 new cd0f7cb KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable. new 766a472 KYLIN-2565 Resolve conflict This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (978bb37) \ N -- N -- N refs/heads/master-hadoop3.1 (766a472) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 6 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml | 8 + KEYS | 117 + LICENSE | 34 +- README.md | 10 +- assembly/pom.xml | 2 +- .../test/java/org/apache/kylin/job/DeployUtil.java | 42 +- .../job/streaming/StreamingTableDataGenerator.java | 5 +- atopcalcite/pom.xml | 7 +- .../org/apache/calcite/runtime/SqlFunctions.java | 2238 ++++++++++++++++++++ .../apache/calcite/sql2rel/SqlToRelConverter.java | 2 +- .../apache/calcite/runtime/SqlFunctionsTest.java | 43 +- ...spark-dependency.sh => check-hive-usability.sh} | 37 +- build/bin/find-hive-dependency.sh | 3 +- build/bin/find-spark-dependency.sh | 2 +- build/bin/kylin-port-replace-util.sh | 2 +- build/bin/kylin.sh | 1 + build/conf/kylin-spark-log4j.properties | 43 + build/conf/setenv.sh | 6 + build/deploy/server.xml | 2 +- build/script/download-tomcat.sh | 6 +- build/script/elimate-jar-conflict.sh | 20 + build/script/prepare.sh | 3 + core-common/pom.xml | 52 +- .../kylin/common/BackwardCompatibilityConfig.java | 20 +- .../java/org/apache/kylin/common/KylinConfig.java | 37 +- .../org/apache/kylin/common/KylinConfigBase.java | 164 +- .../java/org/apache/kylin/common/KylinVersion.java | 2 +- .../java/org/apache/kylin/common/QueryContext.java | 4 +- .../TooBigDictionaryException.java} | 28 +- .../org/apache/kylin/common/htrace/HtraceInit.java | 94 - .../kylin/common/lock/DistributedLockFactory.java | 7 +- .../common/metrics/common/MetricsNameBuilder.java | 8 +- .../common/metrics/metrics2/CodahaleMetrics.java | 3 +- .../metrics/metrics2/JsonFileMetricsReporter.java | 4 +- .../{StorageException.java => BrokenEntity.java} | 46 +- .../common/persistence/BrokenInputStream.java | 58 + .../common/persistence/FileResourceStore.java | 18 +- .../common/persistence/HDFSResourceStore.java | 4 +- .../common/persistence/JDBCConnectionManager.java | 143 ++ .../{StorageException.java => JDBCResource.java} | 54 +- .../kylin/common/persistence/JDBCResourceDAO.java | 707 +++++++ .../common/persistence/JDBCResourceStore.java | 178 ++ .../common/persistence/JDBCSqlQueryFormat.java | 97 + .../persistence/JDBCSqlQueryFormatProvider.java | 54 + .../kylin/common/persistence/ResourceStore.java | 22 +- .../kylin/common/persistence/ResourceTool.java | 33 +- .../common/persistence/RootPersistentEntity.java | 11 +- ...eException.java => WriteConflictException.java} | 9 +- .../apache/kylin/common/restclient/RestClient.java | 32 +- .../kylin/common/util/AbstractApplication.java | 4 +- .../java/org/apache/kylin/common/util/Bytes.java | 96 +- .../org/apache/kylin/common/util/BytesUtil.java | 18 +- .../kylin/common/util/CaseInsensitiveString.java | 4 +- .../org/apache/kylin/common/util/ClassUtil.java | 10 + .../kylin/common/util/CliCommandExecutor.java | 19 +- .../org/apache/kylin/common/util/DateFormat.java | 30 +- .../org/apache/kylin/common/util/DecimalUtil.java | 45 + .../org/apache/kylin/common/util/Dictionary.java | 11 + .../org/apache/kylin/common/util/EncryptUtil.java | 24 +- .../org/apache/kylin/common/util/HadoopUtil.java | 51 + .../apache/kylin/common/util/HiveCmdBuilder.java | 76 +- .../java/org/apache/kylin/common/util/Pair.java | 8 + .../org/apache/kylin/common/util/RandomUtil.java | 21 +- .../org/apache/kylin/common/util/SSHClient.java | 11 +- .../apache/kylin/common/util/SetThreadName.java | 3 +- .../kylin/common/util/SourceConfigurationUtil.java | 112 + .../org/apache/kylin/common/util/StringUtil.java | 3 +- .../org/apache/kylin/common/util/TimeUtil.java | 9 +- .../org/apache/kylin/common/util/ZipFileUtils.java | 93 +- .../src/main/resources/kylin-defaults.properties | 43 +- .../main/resources/metadata-jdbc-mysql.properties | 34 + .../org/apache/kylin/common/KylinConfigTest.java | 52 +- .../common/persistence/ResourceStoreTest.java | 2 +- .../kylin/common/persistence/ResourceToolTest.java | 68 + .../org/apache/kylin/common/util/BasicTest.java | 45 +- .../kylin/common/util/InstallJarIntoMavenTest.java | 12 +- .../apache/kylin/common/util/RandomUtilTest.java | 25 +- .../kylin/common/util/SetAndUnsetSystemProp.java} | 26 +- .../common/util/SourceConfigurationUtilTest.java | 50 + .../org/apache/kylin/common/util/TimeUtilTest.java | 7 +- core-cube/pom.xml | 2 +- .../apache/kylin/cube/CubeCapabilityChecker.java | 38 +- .../org/apache/kylin/cube/CubeDescManager.java | 9 +- .../java/org/apache/kylin/cube/CubeInstance.java | 55 +- .../java/org/apache/kylin/cube/CubeManager.java | 180 +- .../java/org/apache/kylin/cube/CubeSegment.java | 15 +- .../java/org/apache/kylin/cube/CubeUpdate.java | 9 + .../org/apache/kylin/cube/DimensionRangeInfo.java | 113 + .../kylin/cube/cli/DictionaryGeneratorCLI.java | 15 +- .../apache/kylin/cube/common/RowKeySplitter.java | 41 +- .../apache/kylin/cube/common/SegmentPruner.java | 179 ++ .../java/org/apache/kylin/cube/cuboid/Cuboid.java | 2 +- .../apache/kylin/cube/cuboid/CuboidManager.java | 15 + .../apache/kylin/cube/cuboid/CuboidModeEnum.java | 4 +- .../kylin/cube/cuboid/TreeCuboidScheduler.java | 15 +- .../algorithm/AbstractRecommendAlgorithm.java | 24 +- .../cube/cuboid/algorithm/BPUSCalculator.java | 52 +- .../kylin/cube/cuboid/algorithm/BenefitPolicy.java | 21 +- .../cube/cuboid/algorithm/CuboidBenefitModel.java | 144 +- .../kylin/cube/cuboid/algorithm/CuboidStats.java | 37 +- .../cube/cuboid/algorithm/PBPUSCalculator.java | 14 +- .../cube/cuboid/algorithm/SPBPUSCalculator.java | 10 +- .../cuboid/algorithm/generic/BitsChromosome.java | 138 +- .../algorithm/generic/BitsChromosomeHelper.java | 120 ++ .../algorithm/generic/{lib => }/BitsMutation.java | 28 +- ...ntCrossover.java => BitsOnePointCrossover.java} | 55 +- .../generic/CombinedStoppingCondition.java | 4 +- .../cuboid/algorithm/generic/CuboidEncoder.java | 44 - .../cuboid/algorithm/generic/GeneticAlgorithm.java | 235 +- .../algorithm/generic/RouletteWheelSelection.java | 20 +- .../cuboid/algorithm/generic/lib/Chromosome.java | 106 - .../algorithm/generic/lib/ChromosomePair.java | 69 - .../algorithm/generic/lib/CrossoverPolicy.java | 39 - .../generic/lib/ElitisticListPopulation.java | 119 -- .../generic/lib/FixedGenerationCount.java | 73 - .../algorithm/generic/lib/ListPopulation.java | 225 -- .../cuboid/algorithm/generic/lib/Population.java | 61 - .../algorithm/generic/lib/SelectionPolicy.java | 35 - .../algorithm/generic/lib/StoppingCondition.java | 36 - .../algorithm/generic/lib/TournamentSelection.java | 116 - .../cuboid/algorithm/greedy/GreedyAlgorithm.java | 45 +- .../kylin/cube/gridtable/CubeCodeSystem.java | 13 +- .../apache/kylin/cube/gridtable/CubeGridTable.java | 17 +- .../cube/gridtable/CuboidToGridTableMapping.java | 14 +- .../gridtable/CuboidToGridTableMappingExt.java | 134 ++ .../kylin/cube/gridtable/ScanRangePlannerBase.java | 9 +- .../cube/gridtable/TrimmedCubeCodeSystem.java | 4 +- .../kylin/cube/inmemcubing/CuboidResult.java | 8 +- .../inmemcubing/InputConverterUnitForRawData.java | 2 - .../org/apache/kylin/cube/kv/FuzzyMaskEncoder.java | 2 +- .../apache/kylin/cube/kv/RowKeyColumnOrder.java | 108 - .../org/apache/kylin/cube/kv/RowKeyDecoder.java | 12 +- .../org/apache/kylin/cube/kv/RowKeyEncoder.java | 4 +- .../java/org/apache/kylin/cube/model/CubeDesc.java | 115 +- .../kylin/cube/model/CubeJoinedFlatTableDesc.java | 36 +- .../cube/model/CubeJoinedFlatTableEnrich.java | 10 + .../apache/kylin/cube/model/DictionaryDesc.java | 6 +- .../org/apache/kylin/cube/model/DimensionDesc.java | 3 +- .../apache/kylin/cube/model/HBaseMappingDesc.java | 12 +- .../apache/kylin/cube/model/SnapshotTableDesc.java | 75 + .../model/validation/rule/StreamingCubeRule.java | 11 - .../org/apache/kylin/cube/util/CubingUtils.java | 18 +- .../apache/kylin/gridtable/GTAggregateScanner.java | 2 +- .../apache/kylin/gridtable/GTFunctionScanner.java | 117 + .../java/org/apache/kylin/gridtable/GTInfo.java | 17 + .../java/org/apache/kylin/gridtable/GTRecord.java | 12 + .../org/apache/kylin/gridtable/GTScanRequest.java | 79 +- .../kylin/gridtable/GTScanRequestBuilder.java | 41 +- .../java/org/apache/kylin/gridtable/GTUtil.java | 88 +- .../org/apache/kylin/cube/CubeDescManagerTest.java | 78 + .../java/org/apache/kylin/cube/CubeDescTest.java | 12 +- .../org/apache/kylin/cube/CubeInstanceTest.java | 6 +- .../org/apache/kylin/cube/CubeManagerTest.java | 3 + .../apache/kylin/cube/DimensionRangeInfoTest.java | 87 + .../kylin/cube/common/BytesSplitterTest.java | 25 +- .../kylin/cube/common/SegmentPrunerTest.java | 195 ++ .../kylin/cube/cuboid/TreeCuboidSchedulerTest.java | 8 +- .../gridtable/AggregationCacheMemSizeTest.java | 51 +- .../kylin/gridtable/TrimmedCubeCodeSystemTest.java | 46 +- core-dictionary/pom.xml | 7 +- .../apache/kylin/dict/AppendTrieDictionary.java | 46 +- .../org/apache/kylin/dict/DictionaryManager.java | 37 +- .../apache/kylin/dict/GlobalDictionaryBuilder.java | 10 +- .../org/apache/kylin/dict/ShrunkenDictionary.java | 160 ++ .../kylin/dict/ShrunkenDictionaryBuilder.java | 49 + .../java/org/apache/kylin/dict/TrieDictionary.java | 54 + .../apache/kylin/dict/TrieDictionaryBuilder.java | 2 +- .../apache/kylin/dict/TrieDictionaryForest.java | 2 +- .../kylin/dict/TrieDictionaryForestBuilder.java | 17 +- .../apache/kylin/dict/global/AppendDictNode.java | 26 +- .../apache/kylin/dict/global/AppendDictSlice.java | 22 +- .../kylin/dict/global/GlobalDictHDFSStore.java | 18 +- .../dict/global/SegmentAppendTrieDictBuilder.java | 9 +- .../dict/lookup/AbstractLookupRowEncoder.java | 122 ++ .../kylin/dict/lookup/ExtTableSnapshotInfo.java | 149 ++ .../dict/lookup/ExtTableSnapshotInfoManager.java | 213 ++ .../kylin/dict/lookup/IExtLookupProvider.java | 26 +- .../kylin/dict/lookup/IExtLookupTableCache.java | 37 +- .../org/apache/kylin/dict/lookup/ILookupTable.java | 21 +- .../kylin/dict/lookup/LookupProviderFactory.java | 112 + .../kylin/dict/lookup/LookupStringTable.java | 12 +- .../org/apache/kylin/dict/lookup/LookupTable.java | 4 +- .../org/apache/kylin/dict/lookup/SnapshotCLI.java | 4 +- .../apache/kylin/dict/lookup/SnapshotManager.java | 21 +- .../apache/kylin/dict/lookup/SnapshotTable.java | 25 +- .../dict/lookup/cache/RocksDBLookupBuilder.java | 83 + .../dict/lookup/cache/RocksDBLookupRowEncoder.java | 70 + .../dict/lookup/cache/RocksDBLookupTable.java | 113 + .../dict/lookup/cache/RocksDBLookupTableCache.java | 422 ++++ .../kylin/dict/AppendTrieDictionaryTest.java | 62 +- .../apache/kylin/dict/NumberDictionaryTest.java | 20 +- .../apache/kylin/dict/ShrunkenDictionaryTest.java | 77 + .../kylin/dict/TrieDictionaryForestBenchmark.java | 12 +- .../kylin/dict/TrieDictionaryForestTest.java | 5 +- .../org/apache/kylin/dict/TrieDictionaryTest.java | 30 + .../apache/kylin/dict/lookup/LookupTableTest.java | 43 + .../lookup/cache/RocksDBLookupRowEncoderTest.java | 80 + .../lookup/cache/RocksDBLookupTableCacheTest.java | 223 ++ .../dict/lookup/cache/RocksDBLookupTableTest.java | 161 ++ core-job/pom.xml | 2 +- .../java/org/apache/kylin/job/JobInstance.java | 15 + .../java/org/apache/kylin/job/JobSearchResult.java | 66 + .../java/org/apache/kylin/job/JoinedFlatTable.java | 81 +- .../apache/kylin/job/common/PatternedLogger.java | 115 +- .../kylin/job/constant/ExecutableConstants.java | 11 + .../org/apache/kylin/job/dao/ExecutableDao.java | 164 +- .../apache/kylin/job/engine/JobEngineConfig.java | 3 +- .../kylin/job/execution/AbstractExecutable.java | 35 +- .../kylin/job/execution/CheckpointExecutable.java | 7 +- .../job/execution/DefaultChainedExecutable.java | 31 +- .../kylin/job/execution/ExecutableManager.java | 26 +- .../job/impl/threadpool/DefaultFetcherRunner.java | 104 + .../job/impl/threadpool/DefaultScheduler.java | 217 +- .../job/impl/threadpool/DistributedScheduler.java | 123 +- .../kylin/job/impl/threadpool/FetcherRunner.java | 77 + .../kylin/job/impl/threadpool/JobExecutor.java | 18 +- .../job/impl/threadpool/PriorityFetcherRunner.java | 146 ++ .../apache/kylin/job/metrics/JobMetricsFacade.java | 4 +- .../org/apache/kylin/job/SelfStopExecutable.java | 2 +- .../job/impl/threadpool/BaseSchedulerTest.java | 2 +- .../job/impl/threadpool/DefaultSchedulerTest.java | 19 +- core-metadata/pom.xml | 8 +- .../apache/kylin/dimension/DictionaryDimEnc.java | 4 +- .../apache/kylin/dimension/FixedLenHexDimEnc.java | 6 +- .../apache/kylin/measure/MeasureTypeFactory.java | 9 +- .../measure/basic/BigDecimalSumAggregator.java | 9 +- .../kylin/measure/bitmap/BitmapAggregator.java | 10 +- .../kylin/measure/bitmap/BitmapSerializer.java | 14 + .../measure/dim/DimCountDistinctAggregator.java | 5 +- .../extendedcolumn/ExtendedColumnMeasureType.java | 16 +- .../apache/kylin/measure/hllc/HLLCAggregator.java | 5 +- .../apache/kylin/measure/hllc/HLLCSerializer.java | 12 + .../org/apache/kylin/measure/hllc/HLLCounter.java | 4 +- .../measure/percentile/PercentileAggregator.java | 5 +- .../apache/kylin/measure/raw/RawAggregator.java | 6 +- .../apache/kylin/measure/topn/TopNAggregator.java | 8 +- .../org/apache/kylin/measure/topn/TopNCounter.java | 12 +- .../kylin/measure/topn/TopNCounterSerializer.java | 17 +- .../kylin/metadata/TableMetadataManager.java | 15 +- .../kylin/metadata/TempStatementManager.java | 3 +- .../kylin/metadata/cachesync/Broadcaster.java | 6 +- .../kylin/metadata/cachesync/CachedCrudAssist.java | 2 +- .../metadata/datatype/BigDecimalSerializer.java | 10 + .../kylin/metadata/datatype/BooleanSerializer.java | 6 +- .../apache/kylin/metadata/datatype/DataType.java | 41 +- .../kylin/metadata/datatype/DataTypeOrder.java | 155 ++ ...anSerializer.java => DynamicDimSerializer.java} | 51 +- .../metadata/expression/BinaryTupleExpression.java | 149 ++ .../metadata/expression/CaseTupleExpression.java | 180 ++ .../metadata/expression/ColumnTupleExpression.java | 150 ++ .../expression/ExpressionColCollector.java | 120 ++ .../expression/ExpressionCountDistributor.java | 140 ++ .../metadata/expression/ExpressionVisitor.java | 32 +- .../NoneTupleExpression.java} | 43 +- .../metadata/expression/NumberTupleExpression.java | 95 + .../RexCallTupleExpression.java} | 43 +- .../metadata/expression/StringTupleExpression.java | 89 + .../kylin/metadata/expression/TupleExpression.java | 117 + .../expression/TupleExpressionSerializer.java | 239 +++ .../filter/BuiltInFunctionTupleFilter.java | 3 +- .../kylin/metadata/filter/CompareTupleFilter.java | 51 +- .../metadata/filter/FilterCodeSystemFactory.java | 2 + .../apache/kylin/metadata/filter/TupleFilter.java | 116 +- .../metadata/filter/function/BuiltInMethod.java | 5 +- .../kylin/metadata/filter/function/Functions.java | 3 +- .../metadata/filter/function/LikeMatchers.java | 3 +- .../apache/kylin/metadata/model/ColumnDesc.java | 3 +- .../apache/kylin/metadata/model/DataModelDesc.java | 13 +- .../apache/kylin/metadata/model/DatabaseDesc.java | 3 +- .../kylin/metadata/model/DynamicFunctionDesc.java | 85 + .../kylin/metadata/model/ExternalFilterDesc.java | 24 +- .../apache/kylin/metadata/model/FunctionDesc.java | 10 +- .../kylin/metadata/model/IJoinedFlatTableDesc.java | 4 + .../apache/kylin/metadata/model/ISourceAware.java | 4 + .../org/apache/kylin/metadata/model/JoinsTree.java | 2 +- .../kylin/metadata/model/ModelDimensionDesc.java | 3 +- .../apache/kylin/metadata/model/PartitionDesc.java | 16 +- .../metadata/model/SumDynamicFunctionDesc.java | 77 + .../org/apache/kylin/metadata/model/TableDesc.java | 35 +- .../apache/kylin/metadata/model/TableExtDesc.java | 3 +- .../org/apache/kylin/metadata/model/TblColRef.java | 5 +- .../kylin/metadata/model/tool/CalciteParser.java | 3 +- .../kylin/metadata/project/ProjectInstance.java | 14 +- .../kylin/metadata/project/ProjectL2Cache.java | 8 +- .../kylin/metadata/project/ProjectManager.java | 41 +- .../realization/IRealizationConstants.java | 2 + .../kylin/metadata/realization/SQLDigest.java | 21 +- .../kylin/metadata/realization/SQLDigestUtil.java | 137 -- .../main/java/org/apache/kylin/source/ISource.java | 14 +- .../org/apache/kylin/source/SourceFactory.java | 62 - .../org/apache/kylin/source/SourceManager.java | 154 ++ .../source/adhocquery/HivePushDownConverter.java | 94 +- .../source/adhocquery/IPushDownConverter.java | 106 + .../kylin/source/datagen/ColumnGenerator.java | 4 +- .../kylin/source/datagen/ModelDataGenerator.java | 38 +- .../apache/kylin/dimension/IntegerDimEncTest.java | 9 + .../kylin/measure/AggregatorMemEstimateTest.java | 16 +- .../ExtendedColumnSerializerTest.java | 5 +- .../apache/kylin/measure/topn/TopNCounterTest.java | 40 +- .../badquery/BadQueryHistoryManagerTest.java | 8 +- .../datatype/BigDecimalSerializerTest.java | 10 + .../kylin/metadata/datatype/DataTypeOrderTest.java | 57 + .../expression/ExpressionCountDistributorTest.java | 210 ++ .../expression/TupleExpressionSerializerTest.java | 77 + .../metadata/expression/TupleExpressionTest.java | 84 + .../kylin/metadata/filter/TupleFilterTest.java | 77 + .../kylin/metadata/model/FunctionDescTest.java | 60 + core-metrics/pom.xml | 2 +- .../org/apache/kylin/metrics/MetricsManager.java | 3 +- .../apache/kylin/metrics/lib/impl/RecordEvent.java | 3 +- .../metrics/lib/impl/RecordEventTimeDetail.java | 11 +- .../kylin/metrics/lib/impl/TimePropertyEnum.java | 4 +- .../kylin/metrics/property/JobPropertyEnum.java | 4 +- .../metrics/property/QueryCubePropertyEnum.java | 4 +- .../kylin/metrics/property/QueryPropertyEnum.java | 4 +- .../metrics/property/QueryRPCPropertyEnum.java | 4 +- core-storage/pom.xml | 2 +- .../org/apache/kylin/storage/StorageContext.java | 10 + .../storage/gtrecord/CubeScanRangePlanner.java | 86 +- .../kylin/storage/gtrecord/CubeSegmentScanner.java | 22 +- .../kylin/storage/gtrecord/CubeTupleConverter.java | 63 +- .../storage/gtrecord/GTCubeStorageQueryBase.java | 80 +- .../gtrecord/GTCubeStorageQueryRequest.java | 27 +- .../kylin/storage/gtrecord/ITupleConverter.java | 3 +- .../kylin/storage/gtrecord/ScannerWorker.java | 33 +- .../storage/gtrecord/SegmentCubeTupleIterator.java | 3 +- .../gtrecord/SequentialCubeTupleIterator.java | 11 +- .../kylin/storage/hybrid/HybridInstance.java | 19 +- .../kylin/storage/translate/ColumnValueRange.java | 214 -- .../storage/translate/DerivedFilterTranslator.java | 12 +- .../kylin/storage/translate/HBaseKeyRange.java | 273 --- .../kylin/storage/gtrecord/DictGridTableTest.java | 196 +- .../storage/translate/ColumnValueRangeTest.java | 126 -- dev-support/signatures.txt | 25 + engine-mr/pom.xml | 2 +- .../kylin/engine/mr/BatchCubingJobBuilder2.java | 30 +- .../kylin/engine/mr/BatchMergeJobBuilder2.java | 41 +- .../mr/BatchOptimizeJobCheckpointBuilder.java | 3 +- .../java/org/apache/kylin/engine/mr/CubingJob.java | 34 +- .../kylin/engine/mr/ILookupMaterializer.java | 40 + .../java/org/apache/kylin/engine/mr/IMRInput.java | 4 + .../apache/kylin/engine/mr/JobBuilderSupport.java | 136 +- .../kylin/engine/mr/LookupMaterializeContext.java | 84 + .../kylin/engine/mr/LookupSnapshotBuildJob.java | 98 + .../kylin/engine/mr/LookupSnapshotJobBuilder.java | 90 + .../java/org/apache/kylin/engine/mr/MRUtil.java | 25 +- .../kylin/engine/mr/common/AbstractHadoopJob.java | 39 +- .../kylin/engine/mr/common/BaseCuboidBuilder.java | 9 +- .../kylin/engine/mr/common/BatchConstants.java | 14 + .../kylin/engine/mr/common/CubeStatsReader.java | 49 +- .../kylin/engine/mr/common/CubeStatsWriter.java | 16 +- .../engine/mr/common/DictionaryGetterUtil.java | 76 + .../kylin/engine/mr/common/JobInfoConverter.java | 38 +- .../kylin/engine/mr/common/JobRelatedMetaUtil.java | 33 +- .../kylin/engine/mr/common/NDCuboidBuilder.java | 60 +- .../engine/mr/steps/BaseCuboidMapperBase.java | 19 +- .../mr/steps/CalculateStatsFromBaseCuboidJob.java | 1 - .../kylin/engine/mr/steps/CopyDictionaryStep.java | 2 +- .../kylin/engine/mr/steps/CreateDictionaryJob.java | 6 +- .../apache/kylin/engine/mr/steps/CuboidJob.java | 15 +- .../kylin/engine/mr/steps/CuboidReducer.java | 3 +- ...ob.java => ExtractDictionaryFromGlobalJob.java} | 76 +- .../steps/ExtractDictionaryFromGlobalMapper.java | 150 ++ .../mr/steps/FactDistinctColumnPartitioner.java | 5 +- .../engine/mr/steps/FactDistinctColumnsJob.java | 5 +- .../engine/mr/steps/FactDistinctColumnsMapper.java | 160 +- .../mr/steps/FactDistinctColumnsMapperBase.java | 21 +- .../mr/steps/FactDistinctColumnsReducer.java | 115 +- .../steps/FactDistinctColumnsReducerMapping.java | 87 +- .../mr/steps/FilterRecommendCuboidDataJob.java | 7 +- .../mr/steps/FilterRecommendCuboidDataMapper.java | 2 +- .../mr/steps/InMemCuboidFromBaseCuboidJob.java | 4 +- .../kylin/engine/mr/steps/InMemCuboidJob.java | 8 +- .../engine/mr/steps/InMemCuboidMapperBase.java | 15 +- .../kylin/engine/mr/steps/InMemCuboidReducer.java | 3 +- .../kylin/engine/mr/steps/MergeCuboidJob.java | 3 +- .../kylin/engine/mr/steps/MergeCuboidMapper.java | 196 +- .../kylin/engine/mr/steps/MergeDictionaryJob.java | 234 ++ .../engine/mr/steps/MergeDictionaryMapper.java | 207 ++ .../engine/mr/steps/MergeDictionaryReducer.java | 36 +- .../kylin/engine/mr/steps/NDCuboidMapper.java | 5 +- .../kylin/engine/mr/steps/SaveStatisticsStep.java | 9 +- ...ergeCuboidMapper.java => SegmentReEncoder.java} | 185 +- .../engine/mr/steps/SelfDefineSortableKey.java | 11 +- .../kylin/engine/mr/steps/UHCDictionaryJob.java | 4 +- .../engine/mr/steps/UHCDictionaryReducer.java | 3 +- .../mr/steps/UpdateCubeInfoAfterBuildStep.java | 120 +- .../mr/steps/UpdateCubeInfoAfterMergeStep.java | 26 +- .../mr/steps/UpdateCubeInfoAfterOptimizeStep.java | 1 + .../engine/mr/steps/UpdateDictionaryStep.java | 151 ++ .../engine/mr/steps/UpdateOldCuboidShardJob.java | 7 +- .../mr/steps/UpdateOldCuboidShardMapper.java | 9 +- .../mr/steps/lookup/LookupExecutableUtil.java | 134 ++ .../lookup/LookupSnapshotToMetaStoreStep.java | 83 + .../steps/lookup/UpdateCubeAfterSnapshotStep.java | 88 + .../kylin/engine/mr/SortedColumnReaderTest.java | 19 +- .../kylin/engine/mr/steps/CubeReducerTest.java | 9 +- .../kylin/engine/mr/steps/DictColDeduperTest.java | 65 + .../FactDistinctColumnsReducerMappingTest.java | 16 +- .../mr/steps/FactDistinctColumnsReducerTest.java | 4 +- .../mr/steps/LookupMaterializeContextTest.java | 41 + .../engine/mr/steps/SelfDefineSortableKeyTest.java | 3 +- engine-spark/pom.xml | 54 +- .../org/apache/kylin/engine/spark/ISparkInput.java | 34 +- .../apache/kylin/engine/spark/ISparkOutput.java | 129 ++ .../kylin/engine/spark/KylinKryoRegistrator.java | 13 +- .../kylin/engine/spark/KylinSparkJobListener.java | 55 + .../kylin/engine/spark/MultipleOutputsRDD.scala | 114 + .../engine/spark/SparkBatchCubingEngine2.java | 39 +- .../engine/spark/SparkBatchCubingJobBuilder2.java | 117 +- .../engine/spark/SparkBatchMergeJobBuilder2.java | 129 ++ .../kylin/engine/spark/SparkCubingByLayer.java | 184 +- .../kylin/engine/spark/SparkCubingMerge.java | 287 +++ .../apache/kylin/engine/spark/SparkExecutable.java | 414 +++- .../kylin/engine/spark/SparkFactDistinct.java | 906 ++++++++ .../kylin/engine/spark/SparkMergingDictionary.java | 313 +++ .../org/apache/kylin/engine/spark/SparkUtil.java | 178 ++ .../template/cube/kylin_sales_cube.json | 8 +- .../template/cube/kylin_streaming_cube.json | 3 +- .../template/cube_desc/kylin_sales_cube.json | 17 +- .../template/cube_desc/kylin_streaming_cube.json | 2 +- .../template/model_desc/kylin_sales_model.json | 41 +- .../sample_cube/template/project/learn_kylin.json | 4 +- .../table/DEFAULT.KYLIN_ACCOUNT--learn_kylin.json | 4 +- .../table/DEFAULT.KYLIN_CAL_DT--learn_kylin.json | 4 +- ...AULT.KYLIN_CATEGORY_GROUPINGS--learn_kylin.json | 4 +- .../table/DEFAULT.KYLIN_COUNTRY--learn_kylin.json | 4 +- .../table/DEFAULT.KYLIN_SALES--learn_kylin.json | 4 +- .../cube/ssb_cube_with_dimention_range.json | 110 + .../localmeta/cube_desc/ci_left_join_cube.json | 11 +- .../cube_desc/ssb_cube_with_dimention_range.json | 269 +++ examples/test_case_data/localmeta/kylin.properties | 2 +- .../test_case_data/localmeta/kylin_sqoop_conf.xml | 24 + examples/test_case_data/sandbox/kylin.properties | 15 +- external/htrace/pom.xml | 102 - external/pom.xml | 70 - jdbc/pom.xml | 4 +- .../java/org/apache/kylin/jdbc/KylinClient.java | 6 +- .../org/apache/kylin/jdbc/KylinConnection.java | 8 +- .../java/org/apache/kylin/jdbc/DriverTest.java | 23 + kylin-it/pom.xml | 45 +- .../apache/kylin/cube/ITDictionaryManagerTest.java | 10 +- .../cube/cuboid/algorithm/ITAlgorithmTestBase.java | 16 +- .../cuboid/algorithm/ITGeneticAlgorithmTest.java | 46 +- .../kylin/job/BaseTestDistributedScheduler.java | 7 +- .../apache/kylin/job/ContextTestExecutable.java | 44 - .../kylin/job/ITDistributedSchedulerBaseTest.java | 43 +- .../job/ITDistributedSchedulerTakeOverTest.java | 7 +- .../kylin/provision/BuildCubeWithEngine.java | 143 +- .../kylin/provision/BuildCubeWithStream.java | 13 +- .../java/org/apache/kylin/provision/MockKafka.java | 15 +- .../java/org/apache/kylin/query/H2Database.java | 7 +- .../org/apache/kylin/query/ITKylinQueryTest.java | 35 + .../java/org/apache/kylin/query/KylinTestBase.java | 7 +- .../org/apache/kylin/source/SourceManagerTest.java | 61 + .../source/hive/ITHiveSourceTableLoaderTest.java | 4 +- .../kylin/source/hive/ITHiveTezUnionAllTest.java | 108 - .../kylin/source/hive/ITSnapshotManagerTest.java | 6 +- .../source/jdbc/ITJdbcSourceTableLoaderTest.java | 23 +- .../kylin/source/jdbc/ITJdbcTableReaderTest.java | 5 + .../apache/kylin/storage/hbase/ITStorageTest.java | 6 + .../storage/jdbc/ITJDBCResourceStoreTest.java | 319 +++ .../query08.sql => sql/query112.sql} | 11 +- .../query01.sql} | 13 +- .../query02.sql} | 12 +- .../query03.sql} | 12 +- .../query06.sql => sql_expression/query01.sql} | 14 +- .../{sql_timestamp => sql_expression}/query02.sql | 26 +- .../query06.sql => sql_expression/query03.sql} | 18 +- .../query07.sql => sql_expression/query04.sql} | 32 +- .../query07.sql => sql_expression/query05.sql} | 32 +- .../resources/query/sql_expression/query06.sql | 43 + .../{sql_timestamp => sql_ordinal}/query01.sql | 25 +- .../{sql_timestamp => sql_ordinal}/query02.sql | 21 +- .../test/resources/query/sql_timestamp/query01.sql | 5 +- .../test/resources/query/sql_timestamp/query02.sql | 4 +- .../{query01.sql => query02_a.sql.disable} | 7 +- .../sql_timestamp/{query01.sql => query03.sql} | 5 +- .../{query06.sql => query03_b.sql.disable} | 7 +- .../{query01.sql => query03_c.sql.disable} | 8 +- .../{query01.sql => query03_d.sql.disable} | 8 +- .../{query01.sql => query04.sql.disable} | 8 +- .../sql_timestamp/{query01.sql => query05.sql} | 5 +- .../test/resources/query/sql_timestamp/query09.sql | 28 - .../sql_timestamp/{query07.sql => query21.sql} | 4 +- .../sql_timestamp/{query06.sql => query21_a.sql} | 4 +- .../sql_timestamp/{query01.sql => query22.sql} | 5 +- .../sql_timestamp/{query01.sql => query22_a.sql} | 7 +- .../sql_timestamp/{query01.sql => query22_b.sql} | 5 +- .../sql_timestamp/{query01.sql => query22_c.sql} | 5 +- .../sql_timestamp/{query01.sql => query22_d.sql} | 5 +- .../sql_timestamp/{query01.sql => query22_e.sql} | 5 +- .../sql_timestamp/{query01.sql => query23.sql} | 5 +- .../sql_timestamp/{query01.sql => query23_a.sql} | 5 +- .../{query01.sql => query24.sql.disable} | 8 +- .../{query07.sql => query24_a.sql.disable} | 6 +- .../sql_timestamp/{query01.sql => query25.sql} | 5 +- .../sql_timestamp/{query01.sql => query25_a.sql} | 5 +- .../sql_timestamp/{query06.sql => query26.sql} | 17 +- .../src/test/scala/kafka/admin/AdminUtils.scala | 788 +++++++ metrics-reporter-hive/pom.xml | 2 +- .../kylin/metrics/lib/impl/hive/HiveProducer.java | 7 +- metrics-reporter-kafka/pom.xml | 2 +- pom.xml | 1598 +++++++------- query/pom.xml | 2 +- .../org/apache/kylin/query/QueryConnection.java | 1 + .../query/enumerator/DictionaryEnumerator.java | 142 ++ .../kylin/query/enumerator/HiveEnumerator.java | 3 +- .../query/enumerator/LookupTableEnumerator.java | 19 +- .../kylin/query/enumerator/OLAPEnumerator.java | 30 +- .../apache/kylin/query/enumerator/OLAPQuery.java | 5 +- .../query/optrule/AggregateMultipleExpandRule.java | 16 +- .../apache/kylin/query/relnode/ColumnRowType.java | 28 +- .../kylin/query/relnode/OLAPAggregateRel.java | 150 +- .../apache/kylin/query/relnode/OLAPContext.java | 71 +- .../apache/kylin/query/relnode/OLAPFilterRel.java | 299 +-- .../apache/kylin/query/relnode/OLAPJoinRel.java | 33 +- .../apache/kylin/query/relnode/OLAPProjectRel.java | 243 +-- .../apache/kylin/query/relnode/OLAPTableScan.java | 26 + .../apache/kylin/query/relnode/OLAPUnionRel.java | 17 +- .../relnode/visitor/TupleExpressionVisitor.java | 204 ++ .../query/relnode/visitor/TupleFilterVisitor.java | 304 +++ .../org/apache/kylin/query/schema/OLAPSchema.java | 4 + .../kylin/query/schema/OLAPSchemaFactory.java | 28 +- .../org/apache/kylin/query/schema/OLAPTable.java | 25 + .../kylin/query/security/QueryInterceptor.java | 3 +- .../kylin/query/security/QueryInterceptorUtil.java | 8 +- .../apache/kylin/query/security/TableLevelACL.java | 9 +- .../kylin/query/util/DefaultQueryTransformer.java | 5 +- .../kylin/query/util/KeywordDefaultDirtyHack.java | 1 + .../org/apache/kylin/query/util/PushDownUtil.java | 3 +- .../org/apache/kylin/query/util/QueryUtil.java | 10 +- .../java/org/apache/kylin/query/util/RexUtil.java | 57 + .../apache/kylin/query/util/TempStatementUtil.java | 8 +- .../apache/kylin/query/QueryConnectionTest.java | 77 + .../query/util/DefaultQueryTransformerTest.java | 25 +- .../query/util/KeywordDefaultDirtyHackTest.java | 10 + .../org/apache/kylin/query/util/QueryUtilTest.java | 6 + server-base/pom.xml | 6 +- .../kylin/rest/controller/AdminController.java | 3 +- .../kylin/rest/controller/BasicController.java | 15 +- .../kylin/rest/controller/CubeController.java | 148 +- .../rest/controller/ExternalFilterController.java | 4 +- .../kylin/rest/controller/HybridController.java | 47 +- .../kylin/rest/controller/JobController.java | 5 +- .../kylin/rest/controller/ModelController.java | 14 +- .../kylin/rest/controller/ProjectController.java | 7 +- .../kylin/rest/controller/QueryController.java | 6 +- .../kylin/rest/controller/StreamingController.java | 8 +- .../kylin/rest/controller/TableController.java | 59 +- .../rest/exception/TooManyRequestException.java | 58 + .../org/apache/kylin/rest/job/HybridCubeCLI.java | 93 +- .../apache/kylin/rest/job/MetadataCleanupJob.java | 162 ++ .../kylin/rest/job/StorageCleanJobHbaseUtil.java | 130 +- .../apache/kylin/rest/job/StorageCleanupJob.java | 351 +-- .../kylin/rest/metrics/QueryMetricsFacade.java | 16 +- .../rest/request/LookupSnapshotBuildRequest.java | 44 +- .../kylin/rest/request/PrepareSqlRequest.java | 9 + .../kylin/rest/response/CubeInstanceResponse.java | 9 +- .../apache/kylin/rest/response/HybridRespone.java | 61 + .../kylin/rest/response/TableSnapshotResponse.java | 98 + .../kylin/rest/security/AuthoritiesPopulator.java | 7 +- .../rest/security/KylinAclPermissionEvaluator.java | 5 +- .../security/PasswordPlaceholderConfigurer.java | 3 +- .../org/apache/kylin/rest/service/AclService.java | 16 +- .../kylin/rest/service/AclTableMigrationTool.java | 7 +- .../apache/kylin/rest/service/AdminService.java | 8 +- .../kylin/rest/service/BadQueryDetector.java | 5 +- .../org/apache/kylin/rest/service/CubeService.java | 125 +- .../kylin/rest/service/DashboardService.java | 91 +- .../kylin/rest/service/DiagnosisService.java | 9 +- .../apache/kylin/rest/service/EncodingService.java | 12 +- .../kylin/rest/service/ExtFilterService.java | 5 +- .../apache/kylin/rest/service/HybridService.java | 27 +- .../org/apache/kylin/rest/service/JobService.java | 700 ++++-- .../kylin/rest/service/KafkaConfigService.java | 7 +- .../kylin/rest/service/KylinUserService.java | 7 +- .../apache/kylin/rest/service/ModelService.java | 24 +- .../apache/kylin/rest/service/ProjectService.java | 3 +- .../apache/kylin/rest/service/QueryService.java | 371 +++- .../kylin/rest/service/StreamingService.java | 7 +- .../rest/service/TableSchemaUpdateChecker.java | 45 +- .../apache/kylin/rest/service/TableService.java | 310 +-- .../org/apache/kylin/rest/bean/BeanValidator.java | 56 +- .../rest/controller/StreamingControllerTest.java | 33 +- .../kylin/rest/job}/MetadataCleanupJobTest.java | 2 +- .../kylin/rest/job/StorageCleanupJobTest.java | 7 +- .../kylin/rest/service/TableServiceTest.java | 53 + .../src/test/resources/test_meta/UUID | Bin .../test_meta/cube/ci_inner_join_cube.json | 0 .../test_meta/cube_desc/ci_inner_join_cube.json | 0 .../04b6db34-b243-4780-855a-4c7ea4cee817.seq | Bin .../NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict | Bin .../NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict | Bin .../execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930 | 0 .../d861b8b7-c773-47ab-bb1e-c8782ae8d930 | 0 .../d861b8b7-c773-47ab-bb1e-c8782ae8d930-00 | 0 .../d861b8b7-c773-47ab-bb1e-c8782ae8d930-01 | 0 .../src/test/resources/test_meta/kylin.properties | 0 .../test_meta/model_desc/ci_inner_join_model.json | 0 .../test/resources/test_meta/project/default.json | 0 .../test_meta/table/DEFAULT.TEST_ACCOUNT.json | 0 .../table/DEFAULT.TEST_CATEGORY_GROUPINGS.json | 0 .../test_meta/table/DEFAULT.TEST_COUNTRY.json | 0 .../test_meta/table/DEFAULT.TEST_KYLIN_FACT.json | 0 .../test_meta/table/DEFAULT.TEST_ORDER.json | 0 .../resources/test_meta/table/EDW.TEST_CAL_DT.json | 0 .../test_meta/table/EDW.TEST_SELLER_TYPE_DIM.json | 0 .../resources/test_meta/table/EDW.TEST_SITES.json | 0 .../92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot | Bin .../7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot | Bin server/pom.xml | 2 +- server/src/main/resources/kylinSecurity.xml | 2 + .../kylin/rest/controller/CubeControllerTest.java | 24 +- .../kylin/rest/controller/JobControllerTest.java | 19 +- .../kylin/rest/service/AdminServiceTest.java | 58 +- .../kylin/rest/service/BadQueryDetectorTest.java | 4 +- .../kylin/rest/service/CacheServiceTest.java | 6 +- .../kylin/rest/service/KylinQueryTimeoutTest.java | 4 +- .../kylin/rest/service/QueryServiceTest.java | 28 +- .../kylin/rest/service/TableServiceTest.java | 4 +- .../org/apache/kylin/rest/util/AclUtilTest.java | 12 + source-hive/pom.xml | 10 +- .../kylin/source/hive/BeelineHiveClient.java | 15 +- .../apache/kylin/source/hive/CLIHiveClient.java | 12 +- .../kylin/source/hive/CreateFlatHiveTableStep.java | 2 +- .../org/apache/kylin/source/hive/DBConnConf.java | 4 +- .../kylin/source/hive/GarbageCollectionStep.java | 117 + .../apache/kylin/source/hive/HiveInputBase.java | 169 ++ .../org/apache/kylin/source/hive/HiveMRInput.java | 354 +--- .../kylin/source/hive/HiveMetadataExplorer.java | 32 +- .../org/apache/kylin/source/hive/HiveSource.java | 20 +- .../apache/kylin/source/hive/HiveSparkInput.java | 122 ++ .../apache/kylin/source/hive/HiveTableReader.java | 2 +- .../source/hive/RedistributeFlatHiveTableStep.java | 147 ++ .../HiveColumnCardinalityUpdateJob.java | 13 +- .../java/org/apache/kylin/source/jdbc/CmdStep.java | 3 +- .../org/apache/kylin/source/jdbc/JdbcExplorer.java | 47 +- .../apache/kylin/source/jdbc/JdbcHiveMRInput.java | 67 +- .../org/apache/kylin/source/jdbc/JdbcSource.java | 16 +- .../org/apache/kylin/source/jdbc/JdbcTable.java | 9 +- .../apache/kylin/source/jdbc/JdbcTableReader.java | 5 +- .../source/jdbc/metadata/DefaultJdbcMetadata.java | 3 +- .../source/jdbc/metadata/JdbcMetadataFactory.java | 3 +- .../apache/kylin/source/hive/HiveMRInputTest.java | 32 +- .../apache/kylin/source/jdbc/JdbcExplorerTest.java | 5 +- source-kafka/pom.xml | 6 +- .../apache/kylin/source/kafka/KafkaInputBase.java | 163 ++ .../apache/kylin/source/kafka/KafkaMRInput.java | 179 +- .../org/apache/kylin/source/kafka/KafkaSource.java | 69 +- .../apache/kylin/source/kafka/KafkaSparkInput.java | 121 ++ .../kylin/source/kafka/TimedJsonStreamParser.java | 11 +- .../kylin/source/kafka/config/KafkaConfig.java | 12 + .../source/kafka/hadoop/KafkaFlatTableJob.java | 7 +- .../source/kafka/hadoop/KafkaFlatTableMapper.java | 58 +- .../source/kafka/hadoop/KafkaInputFormat.java | 28 +- .../kafka/hadoop/KafkaInputRecordReader.java | 17 +- .../kylin/source/kafka/job/SeekOffsetStep.java | 44 - .../source/kafka/util/KafkaSampleProducer.java | 4 +- storage-hbase/pom.xml | 20 +- .../kylin/storage/hbase/HBaseConnection.java | 3 +- .../kylin/storage/hbase/HBaseResourceStore.java | 7 +- .../apache/kylin/storage/hbase/HBaseStorage.java | 24 +- .../common/coprocessor/CoprocessorProjector.java | 2 +- .../hbase/cube/v2/CubeHBaseEndpointRPC.java | 63 +- .../storage/hbase/cube/v2/CubeHBaseScanRPC.java | 13 +- .../v2/coprocessor/endpoint/CubeVisitService.java | 22 +- .../storage/hbase/lookup/HBaseLookupMRSteps.java | 180 ++ .../hbase/lookup/HBaseLookupMaterializer.java | 40 + .../storage/hbase/lookup/HBaseLookupProvider.java | 58 + .../hbase/lookup/HBaseLookupRowEncoder.java | 134 ++ .../storage/hbase/lookup/HBaseLookupTable.java | 130 ++ .../hbase/lookup/KVSortReducerWithDupKeyCheck.java | 62 + .../hbase/lookup/LookupTableHFilesBulkLoadJob.java | 106 + .../hbase/lookup/LookupTableToHFileJob.java | 292 +++ .../hbase/lookup/LookupTableToHFileMapper.java | 109 + .../UpdateSnapshotCacheForQueryServersStep.java | 106 + .../kylin/storage/hbase/steps/CreateHTableJob.java | 86 +- .../kylin/storage/hbase/steps/CubeHFileJob.java | 26 +- .../kylin/storage/hbase/steps/CubeHTableUtil.java | 50 +- .../storage/hbase/steps/DeprecatedGCStep.java | 198 -- .../{HBaseMRSteps.java => HBaseJobSteps.java} | 98 +- .../kylin/storage/hbase/steps/HBaseMROutput.java | 60 - .../hbase/steps/HBaseMROutput2Transition.java | 46 +- .../kylin/storage/hbase/steps/HBaseMRSteps.java | 245 +-- .../hbase/steps/HBaseSparkOutputTransition.java | 97 + .../kylin/storage/hbase/steps/HBaseSparkSteps.java | 79 + .../kylin/storage/hbase/steps/KeyValueCreator.java | 3 +- .../hbase/steps/RangeKeyDistributionJob.java | 126 -- .../hbase/steps/RangeKeyDistributionMapper.java | 71 - .../hbase/steps/RangeKeyDistributionReducer.java | 133 -- .../kylin/storage/hbase/steps/RowKeyWritable.java | 41 +- .../hbase/steps/SequenceFileCuboidWriter.java | 97 - .../kylin/storage/hbase/steps/SparkCubeHFile.java | 281 +++ .../kylin/storage/hbase/util/CleanHtableCLI.java | 3 +- .../kylin/storage/hbase/util/CubeMigrationCLI.java | 5 +- .../storage/hbase/util/DeployCoprocessorCLI.java | 51 +- .../hbase/util/GridTableHBaseBenchmark.java | 11 +- .../storage/hbase/util/HbaseStreamingInput.java | 14 +- .../apache/kylin/storage/hbase/util/TarGZUtil.java | 69 - .../storage/hbase/util/UpdateHTableHostCLI.java | 7 +- .../storage/hbase/common/HiveJDBCClientTest.java | 12 +- .../hbase/common/coprocessor/FilterBaseTest.java | 8 +- .../hbase/lookup/HBaseLookupRowEncoderTest.java | 98 + .../hbase/lookup/LookupTableToHFileJobTest.java | 110 + ...UpdateSnapshotCacheForQueryServersStepTest.java | 39 +- .../storage/hbase/steps/CubeHFileMapperTest.java | 24 +- .../storage/hbase/steps/HFilePartitionerTest.java | 65 + .../hbase/steps/RangeKeyDistributionJobTest.java | 68 - .../steps/RangeKeyDistributionMapperTest.java | 112 - tomcat-ext/pom.xml | 2 +- tool-assembly/pom.xml | 7 +- tool/pom.xml | 2 +- .../apache/kylin/tool/AbstractInfoExtractor.java | 15 +- .../apache/kylin/tool/AclTableMigrationCLI.java | 7 +- .../org/apache/kylin/tool/CubeMigrationCLI.java | 5 +- .../org/apache/kylin/tool/DiagnosisInfoCLI.java | 2 +- .../apache/kylin/tool/JobInstanceExtractor.java | 7 +- .../org/apache/kylin/tool/KylinLogExtractor.java | 4 +- .../org/apache/kylin/tool/MetadataCleanupJob.java | 155 +- .../metrics/systemcube/CubeInstanceCreator.java | 2 +- .../tool/metrics/systemcube/HiveTableCreator.java | 3 +- .../tool/metrics/systemcube/KylinTableCreator.java | 21 +- .../tool/metrics/systemcube/ModelCreator.java | 2 +- .../kylin/tool/metrics/systemcube/SCCreator.java | 9 +- .../org/apache/kylin/tool/HybridCubeCLITest.java | 48 + .../org/apache/kylin/tool/KylinConfigCLITest.java | 8 +- webapp/Gruntfile.js | 6 + webapp/app/fonts/kylin.eot | Bin 0 -> 1600 bytes webapp/app/fonts/kylin.svg | 13 + webapp/app/fonts/kylin.ttf | Bin 0 -> 1436 bytes webapp/app/fonts/kylin.woff | Bin 0 -> 1512 bytes webapp/app/image/checkbox+.svg | 15 + webapp/app/image/checkbox-.svg | 17 + webapp/app/index.html | 23 + webapp/app/js/controllers/acl.js | 6 +- webapp/app/js/controllers/admin.js | 24 +- webapp/app/js/controllers/cube.js | 4 +- webapp/app/js/controllers/cubeAdvanceSetting.js | 59 + webapp/app/js/controllers/cubeEdit.js | 6 +- webapp/app/js/controllers/cubeMeasures.js | 4 +- webapp/app/js/controllers/cubes.js | 174 +- webapp/app/js/controllers/hybridInstance.js | 110 + webapp/app/js/controllers/hybridInstanceSchema.js | 404 ++++ webapp/app/js/controllers/job.js | 10 +- .../app/js/controllers/modelConditionsSettings.js | 113 +- webapp/app/js/controllers/modelEdit.js | 6 +- webapp/app/js/controllers/models.js | 8 +- webapp/app/js/controllers/page.js | 11 +- webapp/app/js/controllers/projects.js | 4 +- webapp/app/js/controllers/query.js | 2 +- webapp/app/js/controllers/sourceMeta.js | 44 +- webapp/app/js/directives/directives.js | 4 +- webapp/app/js/directives/select.js | 2 +- webapp/app/js/model/cubeConfig.js | 16 +- webapp/app/js/model/hybridInstanceManager.js | 60 + webapp/app/js/model/tableConfig.js | 10 +- webapp/app/js/services/cubes.js | 3 +- .../app/js/services/hybridInstance.js | 27 +- webapp/app/js/services/kylinProperties.js | 7 + .../app/js/services/notify.js | 29 +- webapp/app/js/services/tables.js | 3 +- webapp/app/less/app.less | 127 ++ webapp/app/less/build.less | 1 + webapp/app/less/font.less | 53 + webapp/app/partials/admin/admin.html | 2 +- .../partials/cubeDesigner/advanced_settings.html | 99 + webapp/app/partials/cubes/cubes.html | 2 + webapp/app/partials/cubes/hybrid_edit.html | 176 ++ webapp/app/partials/directives/pagination.html | 2 +- webapp/app/partials/jobs/lookup_refresh.html | 71 + .../modelDesigner/conditions_settings.html | 48 +- webapp/app/partials/models/models_tree.html | 113 +- webapp/app/partials/tables/source_table_tree.html | 8 +- webapp/app/partials/tables/table_detail.html | 56 +- webapp/app/partials/tables/table_load.html | 6 +- webapp/app/routes.json | 14 + 777 files changed, 31550 insertions(+), 10968 deletions(-) create mode 100644 atopcalcite/src/main/java/org/apache/calcite/runtime/SqlFunctions.java copy storage-hbase/src/test/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionReducerTest.java => atopcalcite/src/test/java/org/apache/calcite/runtime/SqlFunctionsTest.java (50%) copy build/bin/{find-spark-dependency.sh => check-hive-usability.sh} (59%) mode change 100755 => 100644 create mode 100644 build/conf/kylin-spark-log4j.properties create mode 100644 build/script/elimate-jar-conflict.sh copy core-common/src/main/java/org/apache/kylin/common/{persistence/StorageException.java => exceptions/TooBigDictionaryException.java} (69%) delete mode 100644 core-common/src/main/java/org/apache/kylin/common/htrace/HtraceInit.java copy core-common/src/main/java/org/apache/kylin/common/persistence/{StorageException.java => BrokenEntity.java} (52%) create mode 100644 core-common/src/main/java/org/apache/kylin/common/persistence/BrokenInputStream.java create mode 100644 core-common/src/main/java/org/apache/kylin/common/persistence/JDBCConnectionManager.java copy core-common/src/main/java/org/apache/kylin/common/persistence/{StorageException.java => JDBCResource.java} (52%) create mode 100644 core-common/src/main/java/org/apache/kylin/common/persistence/JDBCResourceDAO.java create mode 100644 core-common/src/main/java/org/apache/kylin/common/persistence/JDBCResourceStore.java create mode 100644 core-common/src/main/java/org/apache/kylin/common/persistence/JDBCSqlQueryFormat.java create mode 100644 core-common/src/main/java/org/apache/kylin/common/persistence/JDBCSqlQueryFormatProvider.java copy core-common/src/main/java/org/apache/kylin/common/persistence/{StorageException.java => WriteConflictException.java} (84%) create mode 100644 core-common/src/main/java/org/apache/kylin/common/util/DecimalUtil.java copy core-metadata/src/main/java/org/apache/kylin/source/adhocquery/IPushDownConverter.java => core-common/src/main/java/org/apache/kylin/common/util/RandomUtil.java (73%) create mode 100644 core-common/src/main/java/org/apache/kylin/common/util/SourceConfigurationUtil.java create mode 100644 core-common/src/main/resources/metadata-jdbc-mysql.properties create mode 100644 core-common/src/test/java/org/apache/kylin/common/persistence/ResourceToolTest.java copy core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/Fitness.java => core-common/src/test/java/org/apache/kylin/common/util/RandomUtilTest.java (66%) mode change 100755 => 100644 rename core-common/src/{main/java/org/apache/kylin/common/persistence/StorageException.java => test/java/org/apache/kylin/common/util/SetAndUnsetSystemProp.java} (69%) create mode 100644 core-common/src/test/java/org/apache/kylin/common/util/SourceConfigurationUtilTest.java create mode 100644 core-cube/src/main/java/org/apache/kylin/cube/DimensionRangeInfo.java create mode 100644 core-cube/src/main/java/org/apache/kylin/cube/common/SegmentPruner.java create mode 100644 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/BitsChromosomeHelper.java rename core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/{lib => }/BitsMutation.java (70%) rename core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/{lib/OnePointCrossover.java => BitsOnePointCrossover.java} (72%) mode change 100755 => 100644 delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/CuboidEncoder.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/Chromosome.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/ChromosomePair.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/CrossoverPolicy.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/ElitisticListPopulation.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/FixedGenerationCount.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/ListPopulation.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/Population.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/SelectionPolicy.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/StoppingCondition.java delete mode 100755 core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/TournamentSelection.java create mode 100644 core-cube/src/main/java/org/apache/kylin/cube/gridtable/CuboidToGridTableMappingExt.java delete mode 100644 core-cube/src/main/java/org/apache/kylin/cube/kv/RowKeyColumnOrder.java create mode 100644 core-cube/src/main/java/org/apache/kylin/cube/model/SnapshotTableDesc.java create mode 100644 core-cube/src/main/java/org/apache/kylin/gridtable/GTFunctionScanner.java create mode 100644 core-cube/src/test/java/org/apache/kylin/cube/CubeDescManagerTest.java create mode 100644 core-cube/src/test/java/org/apache/kylin/cube/DimensionRangeInfoTest.java create mode 100644 core-cube/src/test/java/org/apache/kylin/cube/common/SegmentPrunerTest.java rename storage-hbase/src/test/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionReducerTest.java => core-cube/src/test/java/org/apache/kylin/gridtable/TrimmedCubeCodeSystemTest.java (51%) create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/ShrunkenDictionary.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/ShrunkenDictionaryBuilder.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/AbstractLookupRowEncoder.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/ExtTableSnapshotInfo.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/ExtTableSnapshotInfoManager.java rename core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/MutationPolicy.java => core-dictionary/src/main/java/org/apache/kylin/dict/lookup/IExtLookupProvider.java (57%) mode change 100755 => 100644 rename core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/ChromosomeMismatchException.java => core-dictionary/src/main/java/org/apache/kylin/dict/lookup/IExtLookupTableCache.java (51%) mode change 100755 => 100644 copy core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/Fitness.java => core-dictionary/src/main/java/org/apache/kylin/dict/lookup/ILookupTable.java (74%) mode change 100755 => 100644 create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/LookupProviderFactory.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupBuilder.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupRowEncoder.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java create mode 100644 core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTableCache.java create mode 100644 core-dictionary/src/test/java/org/apache/kylin/dict/ShrunkenDictionaryTest.java create mode 100644 core-dictionary/src/test/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupRowEncoderTest.java create mode 100644 core-dictionary/src/test/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTableCacheTest.java create mode 100644 core-dictionary/src/test/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTableTest.java create mode 100644 core-job/src/main/java/org/apache/kylin/job/JobSearchResult.java create mode 100644 core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultFetcherRunner.java create mode 100644 core-job/src/main/java/org/apache/kylin/job/impl/threadpool/FetcherRunner.java copy core-metadata/src/main/java/org/apache/kylin/source/adhocquery/IPushDownConverter.java => core-job/src/main/java/org/apache/kylin/job/impl/threadpool/JobExecutor.java (75%) create mode 100644 core-job/src/main/java/org/apache/kylin/job/impl/threadpool/PriorityFetcherRunner.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataTypeOrder.java copy core-metadata/src/main/java/org/apache/kylin/metadata/datatype/{BooleanSerializer.java => DynamicDimSerializer.java} (51%) create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/BinaryTupleExpression.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/CaseTupleExpression.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/ColumnTupleExpression.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/ExpressionColCollector.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/ExpressionCountDistributor.java copy core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/CuboidResult.java => core-metadata/src/main/java/org/apache/kylin/metadata/expression/ExpressionVisitor.java (60%) copy core-metadata/src/main/java/org/apache/kylin/metadata/{datatype/BooleanSerializer.java => expression/NoneTupleExpression.java} (50%) create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/NumberTupleExpression.java copy core-metadata/src/main/java/org/apache/kylin/metadata/{datatype/BooleanSerializer.java => expression/RexCallTupleExpression.java} (50%) create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/StringTupleExpression.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/TupleExpression.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/expression/TupleExpressionSerializer.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/model/DynamicFunctionDesc.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/model/SumDynamicFunctionDesc.java delete mode 100644 core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigestUtil.java delete mode 100644 core-metadata/src/main/java/org/apache/kylin/source/SourceFactory.java create mode 100644 core-metadata/src/main/java/org/apache/kylin/source/SourceManager.java create mode 100644 core-metadata/src/test/java/org/apache/kylin/metadata/datatype/DataTypeOrderTest.java create mode 100644 core-metadata/src/test/java/org/apache/kylin/metadata/expression/ExpressionCountDistributorTest.java create mode 100644 core-metadata/src/test/java/org/apache/kylin/metadata/expression/TupleExpressionSerializerTest.java create mode 100644 core-metadata/src/test/java/org/apache/kylin/metadata/expression/TupleExpressionTest.java create mode 100644 core-metadata/src/test/java/org/apache/kylin/metadata/model/FunctionDescTest.java delete mode 100644 core-storage/src/main/java/org/apache/kylin/storage/translate/ColumnValueRange.java delete mode 100644 core-storage/src/main/java/org/apache/kylin/storage/translate/HBaseKeyRange.java delete mode 100644 core-storage/src/test/java/org/apache/kylin/storage/translate/ColumnValueRangeTest.java create mode 100644 dev-support/signatures.txt create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/ILookupMaterializer.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/LookupMaterializeContext.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/LookupSnapshotBuildJob.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/LookupSnapshotJobBuilder.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DictionaryGetterUtil.java copy engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/{UpdateOldCuboidShardJob.java => ExtractDictionaryFromGlobalJob.java} (61%) create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/ExtractDictionaryFromGlobalMapper.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/MergeDictionaryJob.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/MergeDictionaryMapper.java rename source-kafka/src/main/java/org/apache/kylin/source/kafka/job/UpdateTimeRangeStep.java => engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/MergeDictionaryReducer.java (53%) copy engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/{MergeCuboidMapper.java => SegmentReEncoder.java} (65%) mode change 100755 => 100644 create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/lookup/LookupExecutableUtil.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/lookup/LookupSnapshotToMetaStoreStep.java create mode 100644 engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/lookup/UpdateCubeAfterSnapshotStep.java create mode 100644 engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/DictColDeduperTest.java create mode 100644 engine-mr/src/test/java/org/apache/kylin/engine/mr/steps/LookupMaterializeContextTest.java copy engine-mr/src/main/java/org/apache/kylin/engine/mr/IMRInput.java => engine-spark/src/main/java/org/apache/kylin/engine/spark/ISparkInput.java (66%) create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/ISparkOutput.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/KylinSparkJobListener.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/MultipleOutputsRDD.scala create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchMergeJobBuilder2.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCubingMerge.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkMergingDictionary.java create mode 100644 engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkUtil.java create mode 100644 examples/test_case_data/localmeta/cube/ssb_cube_with_dimention_range.json create mode 100644 examples/test_case_data/localmeta/cube_desc/ssb_cube_with_dimention_range.json create mode 100644 examples/test_case_data/localmeta/kylin_sqoop_conf.xml delete mode 100644 external/htrace/pom.xml delete mode 100644 external/pom.xml delete mode 100644 kylin-it/src/test/java/org/apache/kylin/job/ContextTestExecutable.java create mode 100644 kylin-it/src/test/java/org/apache/kylin/source/SourceManagerTest.java delete mode 100644 kylin-it/src/test/java/org/apache/kylin/source/hive/ITHiveTezUnionAllTest.java create mode 100644 kylin-it/src/test/java/org/apache/kylin/storage/jdbc/ITJDBCResourceStoreTest.java rename kylin-it/src/test/resources/query/{sql_timestamp/query08.sql => sql/query112.sql} (85%) copy kylin-it/src/test/resources/query/{sql_timestamp/query06.sql => sql_dict_enumerator/query01.sql} (62%) copy kylin-it/src/test/resources/query/{sql_timestamp/query06.sql => sql_dict_enumerator/query02.sql} (62%) copy kylin-it/src/test/resources/query/{sql_timestamp/query06.sql => sql_dict_enumerator/query03.sql} (62%) copy kylin-it/src/test/resources/query/{sql_timestamp/query06.sql => sql_expression/query01.sql} (62%) copy kylin-it/src/test/resources/query/{sql_timestamp => sql_expression}/query02.sql (55%) copy kylin-it/src/test/resources/query/{sql_timestamp/query06.sql => sql_expression/query03.sql} (62%) copy kylin-it/src/test/resources/query/{sql_timestamp/query07.sql => sql_expression/query04.sql} (53%) copy kylin-it/src/test/resources/query/{sql_timestamp/query07.sql => sql_expression/query05.sql} (50%) create mode 100644 kylin-it/src/test/resources/query/sql_expression/query06.sql copy kylin-it/src/test/resources/query/{sql_timestamp => sql_ordinal}/query01.sql (59%) copy kylin-it/src/test/resources/query/{sql_timestamp => sql_ordinal}/query02.sql (58%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query02_a.sql.disable} (83%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query03.sql} (88%) copy kylin-it/src/test/resources/query/sql_timestamp/{query06.sql => query03_b.sql.disable} (83%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query03_c.sql.disable} (83%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query03_d.sql.disable} (83%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query04.sql.disable} (83%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query05.sql} (88%) delete mode 100644 kylin-it/src/test/resources/query/sql_timestamp/query09.sql copy kylin-it/src/test/resources/query/sql_timestamp/{query07.sql => query21.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query06.sql => query21_a.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query22.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query22_a.sql} (87%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query22_b.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query22_c.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query22_d.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query22_e.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query23.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query23_a.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query24.sql.disable} (84%) rename kylin-it/src/test/resources/query/sql_timestamp/{query07.sql => query24_a.sql.disable} (84%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query25.sql} (89%) copy kylin-it/src/test/resources/query/sql_timestamp/{query01.sql => query25_a.sql} (89%) rename kylin-it/src/test/resources/query/sql_timestamp/{query06.sql => query26.sql} (62%) create mode 100644 kylin-it/src/test/scala/kafka/admin/AdminUtils.scala create mode 100644 query/src/main/java/org/apache/kylin/query/enumerator/DictionaryEnumerator.java create mode 100644 query/src/main/java/org/apache/kylin/query/relnode/visitor/TupleExpressionVisitor.java create mode 100644 query/src/main/java/org/apache/kylin/query/relnode/visitor/TupleFilterVisitor.java create mode 100644 query/src/main/java/org/apache/kylin/query/util/RexUtil.java create mode 100644 query/src/test/java/org/apache/kylin/query/QueryConnectionTest.java create mode 100644 server-base/src/main/java/org/apache/kylin/rest/exception/TooManyRequestException.java create mode 100644 server-base/src/main/java/org/apache/kylin/rest/job/MetadataCleanupJob.java copy core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/CombinedStoppingCondition.java => server-base/src/main/java/org/apache/kylin/rest/request/LookupSnapshotBuildRequest.java (54%) mode change 100755 => 100644 create mode 100644 server-base/src/main/java/org/apache/kylin/rest/response/HybridRespone.java create mode 100644 server-base/src/main/java/org/apache/kylin/rest/response/TableSnapshotResponse.java rename {tool/src/test/java/org/apache/kylin/tool => server-base/src/test/java/org/apache/kylin/rest/job}/MetadataCleanupJobTest.java (98%) create mode 100644 server-base/src/test/java/org/apache/kylin/rest/service/TableServiceTest.java rename {tool => server-base}/src/test/resources/test_meta/UUID (100%) rename {tool => server-base}/src/test/resources/test_meta/cube/ci_inner_join_cube.json (100%) rename {tool => server-base}/src/test/resources/test_meta/cube_desc/ci_inner_join_cube.json (100%) rename {tool => server-base}/src/test/resources/test_meta/cube_statistics/kylin_sales_cube/04b6db34-b243-4780-855a-4c7ea4cee817.seq (100%) rename {tool => server-base}/src/test/resources/test_meta/dict/DEFAULT.KYLIN_COUNTRY/NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict (100%) rename {tool => server-base}/src/test/resources/test_meta/dict/DEFAULT.TEST_COUNTRY/NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict (100%) rename {tool => server-base}/src/test/resources/test_meta/execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930 (100%) rename {tool => server-base}/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930 (100%) rename {tool => server-base}/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-00 (100%) rename {tool => server-base}/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-01 (100%) rename {tool => server-base}/src/test/resources/test_meta/kylin.properties (100%) rename {tool => server-base}/src/test/resources/test_meta/model_desc/ci_inner_join_model.json (100%) rename {tool => server-base}/src/test/resources/test_meta/project/default.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/DEFAULT.TEST_ACCOUNT.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/DEFAULT.TEST_CATEGORY_GROUPINGS.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/DEFAULT.TEST_COUNTRY.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/DEFAULT.TEST_KYLIN_FACT.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/DEFAULT.TEST_ORDER.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/EDW.TEST_CAL_DT.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/EDW.TEST_SELLER_TYPE_DIM.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table/EDW.TEST_SITES.json (100%) rename {tool => server-base}/src/test/resources/test_meta/table_snapshot/DEFAULT.KYLIN_COUNTRY/92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot (100%) rename {tool => server-base}/src/test/resources/test_meta/table_snapshot/DEFAULT.TEST_COUNTRY/7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot (100%) create mode 100644 source-hive/src/main/java/org/apache/kylin/source/hive/GarbageCollectionStep.java create mode 100644 source-hive/src/main/java/org/apache/kylin/source/hive/HiveInputBase.java create mode 100644 source-hive/src/main/java/org/apache/kylin/source/hive/HiveSparkInput.java create mode 100644 source-hive/src/main/java/org/apache/kylin/source/hive/RedistributeFlatHiveTableStep.java create mode 100644 source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaInputBase.java create mode 100644 source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaSparkInput.java delete mode 100644 source-kafka/src/main/java/org/apache/kylin/source/kafka/job/SeekOffsetStep.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupMRSteps.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupMaterializer.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupProvider.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoder.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupTable.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/KVSortReducerWithDupKeyCheck.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableHFilesBulkLoadJob.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileMapper.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/UpdateSnapshotCacheForQueryServersStep.java delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/DeprecatedGCStep.java copy storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/{HBaseMRSteps.java => HBaseJobSteps.java} (70%) delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseMROutput.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkOutputTransition.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionJob.java delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionMapper.java delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionReducer.java delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/SequenceFileCuboidWriter.java create mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/SparkCubeHFile.java delete mode 100644 storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/TarGZUtil.java create mode 100644 storage-hbase/src/test/java/org/apache/kylin/storage/hbase/lookup/HBaseLookupRowEncoderTest.java create mode 100644 storage-hbase/src/test/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJobTest.java copy query/src/test/java/org/apache/kylin/query/util/KeywordDefaultDirtyHackTest.java => storage-hbase/src/test/java/org/apache/kylin/storage/hbase/lookup/UpdateSnapshotCacheForQueryServersStepTest.java (50%) create mode 100644 storage-hbase/src/test/java/org/apache/kylin/storage/hbase/steps/HFilePartitionerTest.java delete mode 100644 storage-hbase/src/test/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionJobTest.java delete mode 100644 storage-hbase/src/test/java/org/apache/kylin/storage/hbase/steps/RangeKeyDistributionMapperTest.java create mode 100755 webapp/app/fonts/kylin.eot create mode 100755 webapp/app/fonts/kylin.svg create mode 100755 webapp/app/fonts/kylin.ttf create mode 100755 webapp/app/fonts/kylin.woff create mode 100644 webapp/app/image/checkbox+.svg create mode 100644 webapp/app/image/checkbox-.svg create mode 100644 webapp/app/js/controllers/hybridInstance.js create mode 100644 webapp/app/js/controllers/hybridInstanceSchema.js create mode 100644 webapp/app/js/model/hybridInstanceManager.js copy core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/Fitness.java => webapp/app/js/services/hybridInstance.js (59%) mode change 100755 => 100644 rename core-cube/src/main/java/org/apache/kylin/cube/cuboid/algorithm/generic/lib/Fitness.java => webapp/app/js/services/notify.js (68%) mode change 100755 => 100644 create mode 100644 webapp/app/less/font.less create mode 100644 webapp/app/partials/cubes/hybrid_edit.html create mode 100644 webapp/app/partials/jobs/lookup_refresh.html