This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a change to branch kylin-on-parquet-v2
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from 1e64c23 Fix minor bug
new 1b88e90 KYLIN-4147 User has project's admin permission but doesn't
have permission to see the Storage/Planner/streaming tab in Model page
new 4b9e6dc KYLIN-4184 Real time OLAP query gets wrong result
new d3a334d KYLIN-4180 Prevent abnormal CPU usage by limiting flat
filters length
new 3779b11 KYLIN-1716 fix
new 4a00f80 KYLIN-1772 region 0 segment UI refine
new 5d52d3b KYLIN-4168 Add two null judgements before using the object
called input.
new 16ed557 KYLIN-4183 Clicking 'Submit' button is unresponsive, when the
segment is not selected.
new 3bce865 KYLIN-4097: Throw exception when too many dict slice
evictions in AppendTrieDictionary
new 5e1636c Revert "KYLIN-4097: Throw exception when too many dict slice
evictions in AppendTrieDictionary"
new 42ebbd0 KYLIN-3917: Add max segment merge span to cleanup
intermediate data of cube building
new f6a19fa have yarn app url when creating flat hive table job running
new a393abe KYLIN-4169 remove useless listModels() in
DataModelManager.curd.initE… (#843)
new 3a47d37 KYLIN-4175: Support secondary hbase storage config for hbase
cluster migration
new bc79afd fix excepiton while use cube metadata extra validators
new 818b7b0 KYLIN-1024 Upper not support in kylin
new eaf9e3c KYLIN-4197 Fix the problem that no find KYLIN_CONF or
KYLIN_HOME when build dictionary
new 8f3ce86 KYLIN-1850 show kylin version and commit id
new c25639a Bump checkstyle from 8.6 to 8.18
new c8b6b61 KYLIN-4168 fix sonar bug
new 95541aa update docker hbase info port
new 631fd25 KYLIN-4198 “bin/system-cube.sh cron” will overwrite user's
crontab (#893)
new bbfb7fd KYLIN-3906 Fix typos of "ExecutableManger"
new bab187a KYLIN-3741 when the sql result is empty and limit is 0
,should not have 'load more' bar
new 170dd57 KYLIN-4208 RT OLAP kylin.stream.node configure optimization
support all receiver can have the same config
new b49de47 KYLIN-4208 When only the port is configured, change the
program automatically obtains the hostname instead of ip.
new b896222 make sure calcite package is updated automatically
new a80f3fb open source
new f5c7cec open source
new 06911b2 open source
new e978e04 open source
new 5a316f7 open source
new 48fdb27 add test
new aa6d9fa add test
new fbba33d add test
new 7de06ce add test
new 0a4910f open source spark-common
new 81adb34 open source spark-common
new 6310640 open source spark-common
new 0ecf45d add test
new b26770f refactor code
new 847a11b fix maven dependency import error
new 880a684 add new build engine for parquet storage
new 8e79222 add LayoutEntity, IndexEntity and Segment for new metadata
new cf0dffb add dataflow, SegmentRange and new module kylin-spark-job
new 48cea01 resolve conflicts and complete new metadata
new e7d71bd move metadata related classes to metadata.cube.model package
and resolve some dependency problems in class SparkCubingJob
new 7035b1f add local metadata files for new engine spark test
new bf504cb DimensionDesc & MeasureDesc & Cube & DataModel & LayoutEntity
new 9f123a8 remove segment and add path manager
new dba46ed attach new metadata for new spark job builder, add several
properties to KylinConfigBash, update spark executable related interfaces
new 646b71e add spark job build steps(resource detect and cubing)
new 237539d Delete JobTypeEnum.java
new 998b6a3 add contants for spark cubing build
new 074ed80 extract Spanning related metadata
new a3ecc66 metadata dump related before cubing
new 6bd9439 DataLayout & DataSegDetails & DataSegment & SpanningTree
new bc94039 resource detect step before cubing
new 9f996d5 Model revision
new 82b4f82 DataSegment revision
new 9f5e856 DataSegment only handle timestamp by start & end Still using
SegmentRange with suppport only to timestamp in Long & several adds
new 40b11d6 attach new meatadata to DFChooser
new a86aff3 DFBuild related
new c4d3aa2 refine SparkExecutable interface
new 7595c4c Replace metadata in DFBuildJob related classes
new b66175f spark executable related(running on cluster)
new dbc14c4 spark cubing job unit test
new 8e17b1a Fix circle dependency problems
new 453f6dd NGlobalDictBuilderAssist
new f14180c fix forbidden apis problems
new c7e60c5 Add ForestSpanningTree DataModel related modification
MetadataConverter
new e29b8f6 refactor DFChooser and CuboidLayoutChooser related code
new 337aa63 fix dependency problems for cube build and resolve dependency
conflicts
new f00a577 remove CuboidLayoutChoose and add UT
new bbd539d remove non cubing job files
new afb581d fix dependency problems for succeed compile
new 3a61c75 resolve conficts
new da6eef3 refactor CuboidAggregator and add Locale.ROOT to
"toUpperCase".
new af41185 resolve DataType conficts
new 904651d add new metadata to build engine
new 6d397b7 spark cubing job unit test
new a0fc18d remove no use dependency
new c8c1f61 convert layoutEntity and join desc, add UT
new 8a058e7 dump metadata into hdfs, add PathManager for parquet storage
new 81821be normal build success
new 13d754f convert PartitionDesc
new 9c91a14 partitionExp
new 8a5ea1f fix bug with partition column
new 8b3047f remove tmp out files and extract filter condition
new dbbb5ec add count distinct
new 81e345d dump kylin metadata and resources
new a23f8f7 add join selection
new ead5dea add snapshot table
new bcd2694 add UT for checking parquet build result
new 0a1f4ac add new spark build engine interface
new 6ddd16a Update result size into metadata
new 83416ed add hive source
new 33bdcea refactor HiveSource
new 8f35de1 open source
new ec659ee submit parquet cubing job on cluster
new a293b5d parquet spark jar
new 2b0d32c parquet job stats show in UI
new a2ac245 Hide planner if needed
new 2086b2e Add JobTypeEnum Pass CheckStyle plugin Fix cuboid to col bug
new f57cf37 open source
new f998501 open source
new ad2418d open source
new 30f78f7 merge segments for parquet storage build engine
new 1704203 Minor, set url for Kyligence Repository
new a89cd03 Remove old file headers to obey kylin check style rules
new f1e2e9d Support jobs monitor, resume, discard
new 72feb27 Update .gitignore
new 87201a2 fix the problem that kylin-tool.jar package error
new 45e78d0 snapshot ut
new 06a92a1 refactor package name from io.kyligence.kap to
org.apache.kylin
new 167c277 fix dependency problem and update cube after build
new 4eee587 Upgrade lz4 to 1.4.0
new ad28863 Open source spark and roaringbitmap
new 8e89f93 query engine: add minor code
new 2bc169d query engine: storage
new 27bd9d8 query engine: udf
new 1e4481f query engine: udf
new 1b2bb24 query engine : calcite plan to spark plan
new da2ad5e query engine : add olapRel to query context
new 5315466 parquet integration test cases
new dc5896a add new step for update metadata and cleanup for build and
merge
new fe378f9 query engine : completed compile
new 82574ed To pass forbidden API check and UT
new e21f0c3 Upgrade spark version to 2.4.1-os-kylin-r3
new a838f4c Fix compile errors
new 2f5d3f6 query engine : fix class cast exception
new 6bb802e Fix NoClassFound error in cube query
new 61c348b Resolve build problem of integration test and UT
new c7a4db6 query engine : fix ut
new 73f19ba query engine : fix ut
new 15b6717 using local metadata for UT and debug
new f0d12a7 query engine : add file pruning schema
new 7c2ffc3 Refine parquet storage segment info on web
new 4329c80 query engine : add derive
new eebb4cc Set tuple info before generate exec function
new 2be4329 query push down
new 733f632 query engine : add topn, snapshot and pass some query
new 4ec0175 query push down, update file pruner and merge metadata
new c4a47d7 fix the packaging error problem because of the dependency of
org.apache.htrace.Trace
new 5c55895 format scala files header
new 73aa89a Add feature of CSV datasource
new 91cbc07 fix the problem of wrong source records display in front end
new aca4b80 delete unusable metadata files
new 0cf976c fix the problem of file pruning with segment partition and
add integration test case
new 62a0bef Run testing server for curator locally
new 99af9f7 Set scope of curator-test to "provided"
new 7372bfc replace cubeId and segmentId with cubeName and segmentName
for parquet storage path
new 2f815b0 add scalastyle-maven-plugin to check scala code format(file
header, etc.)
new c2464a3 fix
NoClassDefFoundError:org.apache.curator.test.TestingServer when build cube
new 5838569 Refine the script for download spark
new 488f408 Refactor kylin script after drop hbase
new bcea991 KYLIN-4195 The cube size is 'NaN KB' after purging one cube.
new 40a13eb KYLIN-4244: add com.tdunning shade in tool assembly
new 4674d00 KYLIN-4242: [bugfix] Usage instructions in
'PasswordPlaceholderConfigurer' doesn't work
new c350b6a KYLIN-4097: Throw exception when too many dict slice
evictions in AppendTrieDictionary
new 714876e Guava 11 compatibility
new 014f03c Revert "KYLIN-3999, make kylin.query.enable-dynamic-column
default true"
new ac68892 KYLIN-4229 String index out of range -1
new 27984e7 KYLIN-4248 When adding a user, the prompt message is
incorrect when the user name is empty.
new 55bc6da KYLIN-2431 Use StorageCleanupJob cleanup intermediate hive
tables after check whether it was created by current deployment
new fe7a56f KYLIN-4246 Fix for null value and non-exist value in json
parser.
new ca867dc KYLIN-4262 Fix pid in GC filename inconsistent with real pid
new 11ce0ce KYLIN-4201 Allow users to delete unused receivers from
streaming page (#953)
new b010f3b KYLIN-4193 More user-friendly page for loading streaming
tables
new 6b6d4c1 Add more supported date format
new 0a35054 Update BatchMergeJobBuilder2.java (#855)
new 6c77ca9 KYLIN-4178: job scheduler safe mode
new c662418 KYLIN-4162 Add resubmit real time segment REST API
new b520030 KYLIN-4162 Enable resubmitting discarded streaming jobs
new 57313de KYLIN-4172 Enable renaming column name for streaming table
new a8efa10 KYLIN-4205 configuration support set
hive.intermediate-table-prefix
new 2aa437e KYLIN-3842: fix partial match regression in the
kylinProperties.js
new c4d55b6 KYLIN-4190 hiveproducer write() function throw exception
because hive mertics table location path prefix is different with defaut fs
when hdfs uses router-based federation
new 0cb852a fix bug of KYLIN-4178
new 52f8737 KYLIN-4254: The result exporting from Insight with CSV format
is empty, when sql contains Chinese
new d200edd KYLIN-4242 fix
new a5a0766 KYLIN-4187 build dimension dictionary using spark
new 3b0d01d Upgrade calcite to 1.16.0-kylin-r3
new e51e514 KYLIN-1716 UI Location Change Bug Fix
new 98304b4 KYLIN-4257 Build historical data by layer in streaming lambda
cube
new 067f013 KYLIN-4203 Disable a real time cube and then enable it ,this
cube may can't submit build job anymore
new c25f721 KYLIN-4258 Real-time OLAP may return incorrect result for
some case
new 985b83b KYLIN-4273 Make cube planner works for real-time streaming job
new 8e11e4a KYLIN-4265 SQL tab of cube failed when filter is not empty
new 1d7ed60 fix location change url error
new a2b323a [KYLIN-4165] Fix RT OLAP building job on "Save Cube
Dictionaries" step concurrency error (#973)
new fda0e10 KYLIN-4167 Phase2 (#961)
new e7c36bd KYLIN-4283: return SUCCEED while IOException happend in
GarbageCollectionStep
new 1b75f2d Minor, disable spark engine for real time cube
new 5bbb4c0 KYLIN-4292 Use HFileOutputFormat3 in all places to replace
HFileOutputFormat2
new 7e1ee4b Minor, fix typo in KylinHealthCheckJob
new a8b6a2c #4238 Fix kylin_streaming_model broken when changing
kylin.source.hive.database-for-flattable to non-default value
new 5196f75 KYLIN-4280 SegmenPruner support 'OR' and 'NOT'
new ac960cb KYLIN-4287 segmenPruner satisfy "IN" filter bug
new 2aead27 KYLIN-4166 Case when return null when SQL no GROUP BY
new faff746 KYLIN-4226: Skip current unavailable tables when updating
hbase coprocessor
new 8d1f02b KYLIN-4211 PartitionDesc support custom year、month、day
partitions name
new 067729f KYLIN-4211 Add unit test case for
CustomYearMonthDayFieldPartitionConditionBuilder
new 37d7f46 KYLIN-4211 Front end for the feature of setting date
partition in different columns
new 60fae8c KYLIN-4259 fix potential NPE bug reported by FindBugs (#946)
new 2f464c3 KYLIN-4225 close hive session manually
new afe8eac KYLIN-4197 Set deadline for ClientEnvExtractor (#994)
new 3367caa KYLIN-4251 Add livy to docker
new c5cf048 KYLIN-4272: optimize docker build scripts
new 9c465fe KYLIN-4303 Fix the bug that HBaseAdmin is not closed properly
new 2728e47 KYLIN-4304 Project list cannot be correctly sorted by 'Create
Time'
new 5c06ea2 Fix potential RuntimeException
new 558ab4e KYLIN-4252 Fix the error "Cannot read property 'index' of
null" in visualization page
new ff6b5ab KYLIN-2230 can not catch kylin.sh path in linux resouce PATH
setting
new 5056f44 KYLIN-4235 Failed to load table metadata from JDBC data source
new c185862 KYLIN-4212 add user profile page allowing to change password
(#915)
new 7054d05 Minor, change resource URL in Dockerfile
new 8350653 Minor fix for the front end of issue KYLIN-4211
new eb9f62f KYLIN-4308 Make kylin.sh tips clearer and more explicit
new 3eafc56 KYLIN-4237 Return error when execute "explain plan for SQL"
to get the execution plan of SQL
new 821589f KYLIN-3832 Query pushdown support postgresql
new 83217fd KYLIN-3832 Clean up
new 8d3c7f3 KYLIN-3846 Create Flink engine module and initialize module
structure
new bb3f186 KYLIN-3853 Support Flink cube engine for Kylin web UI
new c053e90 KYLIN-3848 Flink cubing step : build by layer
new a9bcef8 KYLIN-3868 Implement Flink batch cubing job builder
new d866ba9 KYLIN-3852 Provide Flink binary package download script
new 34b5cbe KYLIN-3876 Fix flink config option wrong key mapping
new 31f8be6 KYLIN-3850 Flink cubing step : merge by layer
new e4143b2 KYLIN-3877 Add flink specific config items for kylin
properties configuration files
new 96ba199 KYLIN-3879 Implement FlinkEntry
new 9f3d623 KYLIN-3869 Implement Flink batch merge job builder
new fd31bad KYLIN-3897 Implement IFlinkInput based on Hive
new cf0ef3e KYLIN-3896 Implement IFlinkOutput based on HBase
new 35bf9c6 KYLIN-3851 Flink cubing step : merge dictionary
new 468c1bd KYLIN-3910 Fix missing dependency when submit flink
application to YARN
new c58a22d KYLIN-3914 Download required dependencies jar from maven
central repository for FlinkCubeHFile
new ab92818 KYLIN-3919 Add unit test for FlinkOnYarnConfigMapping
new 1fbd6eb Less array copy
new 7576ebf KYLIN-3915 Make HADOOP_CLASSPATH configurable for Flink engine
new 81555e5 KYLIN-3963 Provide a custom serializer for PercentileCounter
new e51da66 KYLIN-3991 Provide more config options for Flink cube engine
new c2875ee KYLIN-4050 Fix a CLI option parse error in
FlinkMergingDictionary and some little code refactor
new ec65240 KYLIN-4071 Fix segments merge with different building
algorithms
new e0f470d Minor, fix merge error during IT test
new 1fcafde KYLIN-4087 Use reduceGroup operator to optimize build cube
step
new a5cbe95 KYLIN-4090 Use mapPartition operator to optimize build cube
step
new 6d4a075 KYLIN-4110 Provider config options to specify yarn queue and
nodelabel
new 5c362f2 KYLIN-4124 Fix bug in map partition function en cuboid
children is empty or null
new 2ff1822 [KYLIN-3921] Bump Flink version from 1.7.1 to 1.9.0
new 3288e47 Fix flink config bug and some minor fix
new 466bb6c Add yarn tracking url in build cube with flink step
new feb0bae KYLIN-4250 FechRunnner should skip the job to process other
jobs instead of throwing exception when the job section metadata is not found
new 402f033 KYLIN-3609 KYLIN-3409 KYLIN-4161 CubeSegmentStatisticsMap
exist only on OLAPQuery.EnumeratorTypeEnum.OLAP query type. (#942)
new 6f69cac KYLIN-4249:DistributedScheduler can selectively assign task
nodes according to cube extra configuration
new 3d1d0bc KYLIN-4240 Kylin SSO with CAS or SAML without LDAP (#958)
new 49f49dc Fix compile error for class MockedDistributedLock
new 4c2174a Fix release test failure
new 795be05 KYLIN-4260 When using server side PreparedStatement cache,
the query result are not match on TopN scenario
new c49e7d9 KYLIN-4311 fix noncompliant interator issue in sonar
new 2fa2acc KYLIN-4249 Fix CI test
new ead858d KYLIN-3237 Not to allow segment deletion by REST API when its
cube is in READY status
new 357ec58 fix bug KYLIN-4300.
new 02d591d KYLIN-4297 Build cube throw NPE when partition column is not
set in JDBC Data Source
new 2687e09 KYLIN-4309 Fixed bug that failed to send to one mailbox will
caused other mailboxes have no chance to send
new fb85423 build uhc dictionary using spark
new 393c1f3 KYLIN-4224 Create flat table wich spark sql
new 4d0d257 KYLIN-4104 Support multi jdbc pushdown runners to execute
query/update
new 4904fb7 fix bug in KYLIN-4243
new e35f7f0 KYLIN-4295 Instances displayed on Query Node are inconsistent
with Job Node
new f68a2ff KYLIN-4302 Fix the bug that InputStream is not closed properly
new fe074ee KYLIN-4271 Use configurable certificate to support LDAPs
authentication of Kylin
new 042a108 KYLIN-4080 Project schema update event casues error reload
NEW DataModelDesc
new 082a5b4 Fix CI test
new 0312041 KYLIN-4193 Support self defined Ts Pattern
new fc80358 Collect information of NumberFormatException
new c686d59 KYLIN-4255 Display detailed error message when using livy
build error
new df027d2 KYLIN-4306: Delete data model desc when encountering
WriteConflictException
new c66b8b1 KYLIN-3956 Segments of not only streaming cube but also batch
cube need to show their status
new bfacd99 KYLIN-4319 in "Extract Dictionary from Global Dictionary"
step, only need load global dictonaries instead of all dictonaries.
new 9c01dd3 delete /execute_output metadata when deleting a job
new 9def975 Fix ci test
new 5152023 KYLIN-4298 KYLIN-4299
new d8057b3 Fix error judgment in PushDownExecutor.java
new 1636fa4 minor, add some ut for dictionaries
new de0cfbb KYLIN-4331 Fix noncompliant method names in child class
new 184de90 KYLIN-4333 Build Server OOM
new c73ac33 making uniform position of variable declaration statements
new e879008 Fix sql injection issue
new 5167dfc Prevent uncontrolled data used in path expression
new 38141f6 Encrept response output for BroadcasterReceiveServlet
new f51de1b Fix "equals" on incomparable types
new 8d6a6d8 Fix type mismatch on container access
new 0c347e7 Fix not thread-safe double-checked locking
new e734621 Fix potential possibility of array index out of bounds error
new a0b0aa6 Remove container never accessed
new f640190 Ensure the validity of http header from concated string
new 5a10f20 Fix synchronization on boxed types or strings
new 4275c52 Fix for KYLIN-4206
new 0660ced KYLIN-4327 TOPN Comparator may violate its general contract
new 8987194 Minor: update json in KylinHealthCheckJob's output
new 6e42b17 Prevent uncontrolled data used in path expression
new 78adff4 Fix equals method does not inspect argument type
new 1ff3ce8 Fix queries built from user-controlled sources
new f2a59e4 Prevent uncontrolled data used in path expression
new 8c6a74b KYLIN-4294: Add http api for metrics
new 6bef250 Prevent uncontrolled data used in path expression
new 590177a KYLIN-4305 Streaming Receiver cannot limit income query
request or cancel long-running query
new 7d0abb1 Update version to 3.1.0-SNAPSHOT
new ce000fd KYLIN-4291 Parallel segment building may causes
WriteConflictException
new b1be18c Validate uuid to prevent sql injection
new fe61009 Revert "KYLIN-4305 Streaming Receiver cannot limit income
query request or cancel long-running query"
new 8f4b2ba Set repo of spring-snapshots with url https
new 4374519 minor, sonar issues for ignoring initial values of method
parameters
new e113cad minor, sonar issues for unchecked stream read
new 158636d minor, sonar issues for throwing exceptions in finnally block
new 5882cb9 minor, sonar issues, volatile variables should not be used
with compound operators
new cce9045 minor, should use BigDecimal.valueOf instead of constructor
to avoid imprecision
new 73aa4f9 KYLIN-4359 Param Value should be required when creating a
cube and adding a new measure
new cb7c654 Minor, throw exception in case of illegal project name
new 5f72032 Minor, add class TableDesc.TableProject to avoid the use of
Pair
new 3dac034 KYLIN-4365 fix RDBMS pushdown with clause bug
new 4b87fd1 KYLIN-4365 Fix sql starts with "with" cannot be queried on
web
new 309ae5e KYLIN-4364 minor,limit varchar length to
DefaultVarcharPrecision
new 64856ef minor,add check to extensible/JdbcExplorer
new f2a33a5 KYLIN-4312 Specified cube when querying by API
new fc971a9 KYLIN-4312 Front end for supporting specified cube when
querying by API
new 5b111ed statement cache eviction invalidation base on time
new 2f6e7ae KYLIN-4334 Auto discard all jobs while disable the Real Time
Cube
new 521f1fc KYLIN-4357: throw Exception when cannot find job (#1073)
new a599814 KYLIN-4349 Close InputStream in RowRecordReader.initReaders()
new 64f9169 KYILN-4328 add cache hold the succeed jobid in job schedule
new 1d46e2a Revert "KYLIN-4349 Close InputStream in
RowRecordReader.initReaders()"
new 560c265 KYLIN-4328 Add code be used to push succeed jobid to hashset.
new 73aee33 KYLIN-4340 Fix bug of get value of isSparkFactDistinctEnable
for cube not correct
new 0e00841 KYLIN-4372 Fix bug of delete file too later cause ZK started
by HBase crash
new 85fbaa3 KYLIN-4377: allow project administrator to operate hybrid
model
new 962fefa KYLIN-4185: optimize CuboidSizeMap by using historical
segments
new fbcb9e8 KYLIN-4314 extend intersect_count and add intersect_value UDAF
new 09305d1 KYLIN-4307 WriteConflictException when build cube by layer
new 87f7ee2 minor, toString() should not return null
new 18bab97 minor, Math operands should be cast before assignment
new 775a003 minor, Non-thread-safe fields should not be static
new e1da3aa minor, compareTo should not be overloaded
new c976036 minor, Iterator.next() methods should throw
NoSuchElementException
new 25cd62a minor, externalizable classes should have no-arguments
constructors
new fd56bbf KYLIN-4336 Global domain dict for MR build engine
new db81a67 KYLIN-4336 Global domain dict for MR build engine ,fix unit
test error
new 57825ef KYLIN-4337 Global domain dict for Spark build engine
new bda32a8 KYLIN-4338 Web page modification to support global domain dict
new 683fcf3 Fix bug for spark build.
new b181e18 KYLIN-4336 Code review
new f979bc7 KYLIN-4383 Fix reflection error
new ad0e17e Use LinkedHashMap for deterministic order
new 974654b upgrade tomcat to 7.0.100
new 9df1c52 KYLIN-4384 Disable create table via spark by default
new 59ae0bd KYLIN-4371 Integrate System Cube with Real-time OLAP (#1102)
new 9b355b1 KYLIN-4353 Add regular check for cube state convert.
new f322f83 KYLIN-4390 Update tomcat to 7.0.100, update server.xml and
web.xml
new 0849ddb KYLIN-4388 Refine the Dockerfile
new fa369da KYLIN-4356: clean outdate DefaultChainedExecutable job
new 5c331e1 KYLIN-4356: add CardinalityExecutable and clean outdate job
new 9709da4 KYLIN-4370 Fix "Table or view not found" error when build
cube with spark for JDBC source at step "Build cube by Layer"
new 4cc49ae KYLIN-4132 Kylin needn't use
"org.apache.directory.api.util.Strings" to import extra dependency
new a88a9ef KYLIN-4389 Change Flink jar dependence from compile to
provided
new b9be241 KYLIN-4387 Flink cubing merge step failed
new f7a10da Set maven for travis
new cb25fd7 update sonar token
new 0c520d9 enable sonar on non-master branch
new dc2b4f3 Skip flink dependency.
new 82249d7 KYLIN-4402 Missing data size in web UI with flink cubing
new 4cae8c5 KYLIN-4400: use client in config to submit sql in
system-cube.sh
new 6dfd0bb KYLIN-4396 Close FileReader in SaveDictStep
new c1f973d Add parameters for sonar check
new 9689b32 Add pull request template
new 6898d55 KYLIN-4426 Refine CliCommandExecutor
new f16c85d add devserver task
new 96013db use system level KylinConfig for migration
new a9cde2e use cube config when uhc step enabled
new 4f85224 KYLIN-4379: use config overwrite for all MR job
new 1092334 KYLIN-4425 Refactor Diagnosis Module
new 63a6763 use newLinkedHashMap for deterministic order
new 060ef06 Check partitionDesc for null value before use it
new fd0c42e KYLIN-4399 Redirect http url is not override as https when
using vip
new f140226 Set version to 4.0.0-SNAPSHOT
new fd10a09 Make column name writable for csv data source
new 952407c Hide all build engine except Spark2
new ec6e7de Add link to Development-document
new 4d5a234 Enable cubing locally
new c59d172 fix query error because of file pruner wrong segment path
new 80e88ae #138 Fix query full build cube got NullPointerException
new 1048f44 #137 Fix cannot read csv source if Partition Date Column is
set in the model
new c3a83a9 #141 Simplify parquet local metadata
new 49509bd #136 #83 Pass unit test cases and integration test cases
new ef68fce #153 Fix license check failed
new fc7fc28 #153 Add license headers
new a12d96c Add workflow file
new a61d213 open source
new 0053580 add test
new 99df0b1 add test
new 5354f55 add test
new 4321c2e add test
new 0c6df26 #145 Add unit test cases for spark cubing engine
new 713b7d5 #153 Add license header
new f341806 #153 Add license header
new 99e9633 #158 Replace maven repo url with https
new 0e88d9f #157 Fix query on refreshed cube failed with
FileNotFoundException
new 6396252 query engine : add spark classloader
new bf9d2d1 #155 fix class loader problem when query
new ed42c3f #157 Simplify parquet storage path
new cd22a0a Minor, default class for pushdwon
new 7e78f2b #161 Fix query failed problem on HDP and remove tomcat-ext
module
new 958f73a #167 No data output from step ResourceDetectBeforeCubingJob
with binary package
new a1fca74 #173 Will get direct parent and ancestor cuboids with method
findDirectParentCandidates
new 36337ae KYLIN-4444 Fix segment refresh error in parquet storage
new 73fe07d KYLIN-4463 Pass ignored test cases
new aebc7fa KYLIN-4463 Pass ignored test cases
new 34ac411 KYLIN-4462 Support count distinct,TopN,Percentile
new b0385e2 KYLIN-4454 Fix query on snapshot table failed
new 90391f5 KYLIN-4463 Fix unstable ut cases
new 48310a5 KYLIN-4455 Query will fail when set calcite.debug=true
new 8a425b2 KYLIN-4467 Support TopN by kylin on Parquet
new 0d06367 KYLIN-4451 Fix ClassCastException when querying on cluster
with binary package
new db6650e KYLIN-4469 Fix clone model error
new e81191e KYLIN-4466 Fix cannot unload table which is loaded from CSV
source
new 243ec8e KYLIN-4474 Fix unsupported sql of window function in IT test
new 54e51a3 KYLIN-4475 Add intersec count test cases in IT
new 734a003 KYLIN-4483 Avoid to build global dictionaries with empty
ColumnDesc collection
new 073300f KYLIN-4482 Too many logging segment info with CubeBuildJob
step
new 2ed1d08 KYLIN-4450 Add the feature that adjusting spark driver memory
adaptively
new 334097e KYLIN-4456 Temporary files generated by UT or Integration
Tests need to be deleted
new bdb52b3 KYLIN-4458 FilePruner prune shards
new 4054206 KYLIN-4449 A running build job will still running when cancel
from front end
new ec8a137 KYLIN-4542 After downloading spark with bin/download-spark.sh
, still need set SPARK_HOME manually .
The 433 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/pull_request_template.md | 28 +
.github/workflows/maven.yml | 39 +
.gitignore | 7 +
.travis.yml | 10 +-
README.md | 4 +
assembly/pom.xml | 7 +-
build/bin/build-incremental-cube.sh | 2 +-
build/bin/check-env.sh | 10 +-
build/bin/check-hive-usability.sh | 2 +-
build/bin/check-migration-acl.sh | 9 +-
build/bin/check-port-availability.sh | 4 +-
build/bin/diag.sh | 19 +-
build/bin/download-flink.sh | 81 +
build/bin/download-spark.sh | 18 +-
build/bin/find-flink-dependency.sh | 52 +
build/bin/find-hadoop-conf-dir.sh | 6 +-
build/bin/find-hbase-dependency.sh | 20 +-
build/bin/find-hive-dependency.sh | 2 +-
build/bin/find-kafka-dependency.sh | 2 +-
build/bin/find-spark-dependency.sh | 14 +-
build/bin/get-properties.sh | 4 +-
build/bin/header.sh | 17 +
build/bin/kylin-port-replace-util.sh | 4 +-
build/bin/kylin.sh | 252 +-
build/bin/load-hive-conf.sh | 2 +-
build/bin/metastore.sh | 2 +-
build/bin/sample-streaming.sh | 2 +-
build/bin/sample.sh | 15 +-
build/bin/system-cube.sh | 47 +-
build/conf/kylin-server-log4j.properties | 24 +-
build/conf/kylin_job_conf_cube_merge.xml | 104 +
build/conf/setenv.sh | 4 +-
build/deploy/context.xml | 2 +-
build/deploy/server.xml | 7 +-
build/script/compress.sh | 4 +-
build/script/download-flink.sh | 71 +
build/script/download-spark.sh | 56 +-
build/script/download-tomcat.sh | 17 +-
build/script/package.sh | 1 +
build/script/prepare-libs.sh | 1 +
cache/pom.xml | 2 +-
.../spy/memcached/RefinedKetamaNodeLocator.java | 25 +-
core-common/pom.xml | 7 +-
.../org/apache/kylin/common/JobProcessContext.java | 40 +
.../java/org/apache/kylin/common/KylinConfig.java | 82 +-
.../org/apache/kylin/common/KylinConfigBase.java | 702 +-
.../org/apache/kylin/common/KylinConfigExt.java | 2 +
.../java/org/apache/kylin/common/KylinVersion.java | 6 +-
.../java/org/apache/kylin/common/QueryContext.java | 80 +-
.../org/apache/kylin/common/SourceDialect.java | 8 +-
.../java/org/apache/kylin/common/StorageURL.java | 2 +-
.../apache/kylin/common/debug/BackdoorToggles.java | 12 +
.../apache/kylin/common/livy/LivyRestExecutor.java | 52 +-
.../apache/kylin/common/lock/DistributedLock.java | 9 +-
.../common/metrics/metrics2/CodahaleMetrics.java | 13 +-
.../persistence/ExponentialBackoffRetry.java | 2 +-
.../common/persistence/HDFSResourceStore.java | 2 +-
.../common/persistence/JDBCResourceStore.java | 47 +-
.../kylin/common/persistence/ResourceStore.java | 7 +-
.../common/persistence/RootPersistentEntity.java | 9 +-
.../common/storage/DefaultStorageProvider.java | 40 +
.../kylin/common/storage/IStorageProvider.java | 29 +
.../org/apache/kylin/common/util/ByteArray.java | 2 +-
.../kylin/common/util/CliCommandExecutor.java | 131 +-
.../org/apache/kylin/common/util/DateFormat.java | 22 +-
.../org/apache/kylin/common/util/DecimalUtil.java | 7 +-
.../org/apache/kylin/common/util/FIFOIterator.java | 4 +
.../org/apache/kylin/common/util/FileUtils.java | 35 +
.../org/apache/kylin/common/util/FlinkEntry.java | 48 +
.../org/apache/kylin/common/util/HadoopUtil.java | 47 +-
.../apache/kylin/common/util/ImmutableBitSet.java | 28 +
.../org/apache/kylin/common/util/JsonUtil.java | 35 +
.../org/apache/kylin/common/util/MailService.java | 23 +-
.../org/apache/kylin/common/util/TimeUtil.java | 6 +-
.../org/apache/kylin/common/util/ToolUtil.java | 116 +
.../org/apache/kylin/common/util/VersionUtil.java | 62 +
.../java/org/apache/kylin/common/util/ZKUtil.java | 48 +-
.../src/main/resources/kylin-defaults.properties | 75 +-
.../kylin/common/util/CliCommandExecutorTest.java | 42 +
.../common/util/LocalFileMetadataTestCase.java | 23 +-
.../kylin/common/util/TempMetadataBuilder.java | 200 +
.../org/apache/kylin/common/util/ToolUtilTest.java | 41 +
core-cube/pom.xml | 2 +-
.../java/org/apache/kylin/cube/CubeManager.java | 58 +-
.../java/org/apache/kylin/cube/CubeSegment.java | 28 +
.../apache/kylin/cube/common/SegmentPruner.java | 56 +-
.../apache/kylin/cube/common/TupleFilterNode.java | 76 +
.../cube/cuboid/algorithm/BPUSCalculator.java | 2 +-
.../cube/cuboid/algorithm/CuboidStatsUtil.java | 4 +-
.../gridtable/CuboidToGridTableMappingExt.java | 4 +-
.../RecordConsumeBlockingQueueController.java | 18 +-
.../java/org/apache/kylin/cube/model/CubeDesc.java | 34 +-
.../CubeDescTiretreeGlobalDomainDictUtil.java | 170 +
.../kylin/cube/model/CubeJoinedFlatTableDesc.java | 7 +-
.../apache/kylin/cube/model/DictionaryDesc.java | 42 +
.../model/validation/CubeMetadataValidator.java | 3 +-
.../cube/model/validation/rule/DictionaryRule.java | 12 +-
.../model/validation/rule/StreamingCubeRule.java | 3 +-
.../apache/kylin/gridtable/GTAggregateScanner.java | 18 +-
.../org/apache/kylin/cube/CubeManagerTest.java | 234 +-
.../kylin/cube/common/SegmentPrunerTest.java | 176 +
core-dictionary/pom.xml | 8 +-
.../apache/kylin/dict/AppendTrieDictionary.java | 48 +-
.../kylin/dict/BuiltInFunctionTransformer.java | 21 +-
.../org/apache/kylin/dict/DictionaryManager.java | 31 +-
.../org/apache/kylin/dict/NumberDictionary.java | 10 +-
.../org/apache/kylin/dict/NumberDictionary2.java | 4 +-
.../org/apache/kylin/dict/TimeStrDictionary.java | 5 +-
.../apache/kylin/dict/lookup/SnapshotManager.java | 64 +-
.../dict/lookup/cache/RocksDBLookupTableCache.java | 46 +-
.../kylin/dict/AppendTrieDictionaryTest.java | 114 +-
.../apache/kylin/dict/DateStrDictionaryTest.java | 102 +
.../apache/kylin/dict/ShrunkenDictionaryTest.java | 105 +-
.../apache/kylin/dict/TimeStrDictionaryTest.java | 74 +-
core-job/pom.xml | 2 +-
.../java/org/apache/kylin/job/JobInstance.java | 10 +
.../java/org/apache/kylin/job/JoinedFlatTable.java | 70 +-
.../java/org/apache/kylin/job/JoinedFormatter.java | 2 +-
.../apache/kylin/job/common/PatternedLogger.java | 11 +-
.../apache/kylin/job/common/ShellExecutable.java | 2 +-
.../kylin/job/constant/ExecutableConstants.java | 12 +
.../org/apache/kylin/job/dao/ExecutableDao.java | 37 +-
.../apache/kylin/job/engine/JobEngineConfig.java | 1 +
.../kylin/job/exception/JobStoppedException.java | 27 +
.../kylin/job/execution/AbstractExecutable.java | 140 +-
.../kylin/job/execution/CardinalityExecutable.java | 29 +
.../kylin/job/execution/ChainedExecutable.java | 1 +
.../job/execution/DefaultChainedExecutable.java | 35 +
.../kylin/job/execution/ExecutableManager.java | 36 +-
.../kylin/job/execution/ExecutableState.java | 4 +
.../apache/kylin/job/execution/ExecuteResult.java | 13 +
.../apache/kylin/job/execution/JobTypeEnum.java | 25 +
.../job/impl/threadpool/DefaultFetcherRunner.java | 36 +-
.../job/impl/threadpool/DistributedScheduler.java | 240 +-
.../kylin/job/impl/threadpool/FetcherRunner.java | 10 +-
.../job/impl/threadpool/PriorityFetcherRunner.java | 36 +-
.../lock/zookeeper/ZookeeperDistributedLock.java | 28 +
.../kylin/job/lock/zookeeper/ZookeeperJobLock.java | 5 +
.../apache/kylin/job/metrics/JobMetricsFacade.java | 9 +-
.../kylin/job/util/FlatTableSqlQuoteUtils.java | 103 +-
.../org/apache/kylin/job/JoinedFormatterTest.java | 29 +
.../job/impl/threadpool/DefaultSchedulerTest.java | 14 -
.../kylin/job/util/FlatTableSqlQuoteUtilsTest.java | 14 +-
core-metadata/pom.xml | 2 +-
.../apache/kylin/dimension/AbstractDateDimEnc.java | 4 +
.../kylin/dimension/TimeDerivedColumnType.java | 18 +
.../BitmapIntersectDistinctCountAggFunc.java | 48 +-
.../bitmap/BitmapIntersectValueAggFunc.java | 54 +
.../kylin/measure/bitmap/BitmapMeasureType.java | 15 +-
.../measure/bitmap/RetentionPartialResult.java | 120 +
.../apache/kylin/measure/hllc/HLLCAggregator.java | 6 +
.../measure/percentile/PercentileSerializer.java | 5 +-
.../kylin/measure/topn/DoubleDeltaSerializer.java | 6 +-
.../org/apache/kylin/measure/topn/TopNCounter.java | 14 +-
.../apache/kylin/measure/topn/TopNMeasureType.java | 7 +-
.../apache/kylin/metadata/MetadataConstants.java | 45 +-
.../kylin/metadata/TableMetadataManager.java | 15 +-
.../metadata/badquery/BadQueryHistoryManager.java | 2 +
.../metadata/datatype/DataTypeSerializer.java | 5 +-
.../apache/kylin/metadata/filter/TupleFilter.java | 13 +-
.../apache/kylin/metadata/model/CsvColumnDesc.java | 83 +
.../apache/kylin/metadata/model/DataModelDesc.java | 42 +-
.../kylin/metadata/model/DataModelManager.java | 40 +-
.../apache/kylin/metadata/model/FunctionDesc.java | 4 +-
.../apache/kylin/metadata/model/IEngineAware.java | 3 +
.../apache/kylin/metadata/model/ISourceAware.java | 1 +
.../apache/kylin/metadata/model/IStorageAware.java | 1 +
.../apache/kylin/metadata/model/PartitionDesc.java | 69 +-
.../apache/kylin/metadata/model/SegmentRange.java | 8 +
.../org/apache/kylin/metadata/model/Segments.java | 25 +-
.../org/apache/kylin/metadata/model/TableDesc.java | 48 +-
.../apache/kylin/metadata/model/TableExtDesc.java | 137 +-
.../org/apache/kylin/metadata/model/TableRef.java | 6 +-
.../org/apache/kylin/metadata/model/TblColRef.java | 5 +
.../kylin/metadata/realization/SQLDigest.java | 4 +-
.../org/apache/kylin/metadata/tuple/Tuple.java | 2 +-
.../kylin/source/datagen/ColumnGenerator.java | 12 +-
.../kylin/measure/topn/TopNCounterBasicTest.java | 47 +
.../kylin/metadata/model/DataModelManagerTest.java | 7 -
.../kylin/metadata/model/PartitionDescTest.java | 61 +
core-metrics/pom.xml | 2 +-
.../org/apache/kylin/metrics/MetricsManager.java | 2 +-
.../apache/kylin/metrics/lib/ActiveReservoir.java | 15 +-
.../kylin/metrics/lib/ActiveReservoirFilter.java | 44 -
.../metrics/lib/ActiveReservoirRecordFilter.java | 53 +
.../kylin/metrics/lib/ActiveReservoirReporter.java | 4 +-
.../java/org/apache/kylin/metrics/lib/Record.java | 6 +-
.../java/org/apache/kylin/metrics/lib/Sink.java | 2 +-
.../org/apache/kylin/metrics/lib/SinkTool.java | 32 -
.../metrics/lib/impl/BaseScheduledReporter.java | 3 +-
.../kylin/metrics/lib/impl/BlockingReservoir.java | 9 +-
.../kylin/metrics/lib/impl/InstantReservoir.java | 8 +-
.../kylin/metrics/lib/impl/MetricsSystem.java | 8 +-
.../apache/kylin/metrics/lib/impl/RecordEvent.java | 57 +-
.../metrics/lib/impl/RecordEventTimeDetail.java | 5 +-
.../kylin/metrics/lib/impl/ReporterBuilder.java | 8 +-
.../kylin/metrics/lib/impl/TimePropertyEnum.java | 11 +-
.../kylin/metrics/property/JobPropertyEnum.java | 1 +
.../kylin/metrics/property/QueryPropertyEnum.java | 1 +
core-storage/pom.xml | 2 +-
.../org/apache/kylin/storage/StorageContext.java | 2 +-
.../kylin/storage/gtrecord/CubeTupleConverter.java | 17 +-
.../storage/gtrecord/GTCubeStorageQueryBase.java | 5 +-
.../gtrecord/SortedIteratorMergerWithLimit.java | 3 +-
.../kylin/storage/gtrecord/DictGridTableTest.java | 2 +-
.../kylin/storage/hybrid/HybridInstanceTest.java | 1 +
datasource-sdk/pom.xml | 8 +-
.../datasource/adaptor/AbstractJdbcAdaptor.java | 4 +-
.../sdk/datasource/adaptor/PostgresqlAdaptor.java | 107 +
.../framework/FixedCachedRowSetImpl.java | 106 +-
.../sdk/datasource/framework/JdbcConnector.java | 4 +-
.../framework/SourceConnectorFactory.java | 9 +-
.../datasource/framework/conv/ConvSqlWriter.java | 8 +-
.../framework/conv/DefaultConfiguer.java | 138 -
.../framework/conv/DefaultConfigurer.java | 137 +
.../framework/conv/GenericSqlConverter.java | 2 +-
.../datasource/framework/conv/SqlConverter.java | 2 +-
.../src/main/resources/datasource/postgresql.xml | 78 +
.../framework/conv/GenericSqlConverterTest.java | 71 -
.../framework/conv/PostgresqlSqlConverterTest.java | 94 +
docker/Dockerfile | 103 +-
docker/Dockerfile_dev | 43 +
docker/Dockerfile_hadoop | 105 +
docker/README.md | 145 +
docker/build_image.sh | 22 +-
docker/conf/hbase/hbase-site.xml | 2 +-
docker/entrypoint.sh | 19 +-
docker/run_container.sh | 3 +-
docker/use-guide.md | 136 -
engine-flink/pom.xml | 158 +
.../engine/flink/FlinkBatchCubingEngine2.java | 68 +
.../engine/flink/FlinkBatchCubingJobBuilder2.java | 125 +
.../engine/flink/FlinkBatchMergeJobBuilder2.java | 132 +
.../kylin/engine/flink/FlinkCubingByLayer.java | 636 ++
.../kylin/engine/flink/FlinkCubingMerge.java | 352 +
.../apache/kylin/engine/flink/FlinkExecutable.java | 447 +
.../kylin/engine/flink/FlinkMergingDictionary.java | 309 +
.../engine/flink/FlinkOnYarnConfigMapping.java | 83 +
.../org/apache/kylin/engine/flink/FlinkUtil.java | 133 +
.../org/apache/kylin/engine/flink/IFlinkInput.java | 60 +
.../apache/kylin/engine/flink/IFlinkOutput.java | 132 +
.../flink/util/PercentileCounterSerializer.java | 67 +
.../engine/flink/FlinkOnYarnConfigMappingTest.java | 109 +
engine-mr/pom.xml | 2 +-
.../kylin/engine/mr/BatchMergeJobBuilder2.java | 8 +-
.../mr/BatchOptimizeJobCheckpointBuilder.java | 2 +-
.../apache/kylin/engine/mr/ByteArrayWritable.java | 7 +-
.../java/org/apache/kylin/engine/mr/CubingJob.java | 84 +-
.../apache/kylin/engine/mr/JobBuilderSupport.java | 46 +
.../kylin/engine/mr/StreamingCubingJobBuilder.java | 23 +-
.../kylin/engine/mr/common/AbstractHadoopJob.java | 4 +
.../kylin/engine/mr/common/BatchConstants.java | 3 +
.../kylin/engine/mr/common/CubeJobLockUtil.java | 46 +
.../kylin/engine/mr/common/CubeStatsReader.java | 99 +-
.../engine/mr/common/CuboidStatsReaderUtil.java | 3 +-
.../kylin/engine/mr/common/JobInfoConverter.java | 1 +
.../engine/mr/common/MapReduceExecutable.java | 256 +-
.../engine/mr/common/StatisticsDecisionUtil.java | 4 +
.../kylin/engine/mr/steps/CreateDictionaryJob.java | 4 +-
.../steps/ExtractDictionaryFromGlobalMapper.java | 10 +-
.../mr/steps/UpdateCubeInfoAfterBuildStep.java | 5 +
.../mr/steps/UpdateCubeInfoAfterMergeStep.java | 5 +
.../engine/mr/steps/UpdateDictionaryStep.java | 2 +-
.../kylin/engine/mr/streaming/ColumnToRowJob.java | 2 +-
.../mr/streaming/ColumnarSplitDictReader.java | 2 +-
.../kylin/engine/mr/streaming/DictsReader.java | 2 +
.../kylin/engine/mr/streaming/SaveDictStep.java | 127 +-
engine-spark/pom.xml | 2 +-
.../kylin/engine/spark/KylinKryoRegistrator.java | 1 +
.../engine/spark/SparkBatchCubingJobBuilder2.java | 54 +-
.../kylin/engine/spark/SparkBuildDictionary.java | 500 +
.../kylin/engine/spark/SparkCreatingFlatTable.java | 43 +
.../kylin/engine/spark/SparkCubingByLayer.java | 2 +-
.../apache/kylin/engine/spark/SparkExecutable.java | 68 +-
.../kylin/engine/spark/SparkExecutableLivy.java | 28 +-
.../kylin/engine/spark/SparkFactDistinct.java | 10 +-
.../apache/kylin/engine/spark/SparkSqlBatch.java | 131 +
.../kylin/engine/spark/SparkUHCDictionary.java | 321 +
.../org/apache/kylin/engine/spark/SparkUtil.java | 4 +-
.../spark/util/PercentileCounterSerializer.java | 11 +-
.../file_prunning/cube/file_pruning_cube.json | 17 +
.../file_prunning/cube/file_pruning_cube2.json | 17 +
.../cube/file_pruning_cube_measure.json | 17 +
.../file_prunning/cube_desc/file_pruning_cube.json | 135 +
.../cube_desc/file_pruning_cube2.json | 122 +
.../cube_desc/file_pruning_cube_measure.json | 67 +
.../file_prunning/data/DEFAULT.TEST_KYLIN_FACT.csv | 10000 +++++++++++++++++++
.../file_prunning/data/DEFAULT.TEST_MEASURE.csv | 15 +
.../data/DEFAULT.TEST_ORDER.csv | 0
.../test_case_data/file_prunning/kylin.properties | 157 +
.../model_desc/file_pruning_model.json | 40 +
.../model_desc/file_pruning_model2.json | 40 +
.../model_desc/file_pruning_model_measure.json | 28 +
.../file_prunning/project/default.json | 25 +
.../table/DEFAULT.TEST_KYLIN_FACT.json | 74 +
.../file_prunning/table/DEFAULT.TEST_MEASURE.json | 112 +
.../file_prunning/table/DEFAULT.TEST_ORDER.json | 36 +
.../localmeta/cube/ci_inner_join_cube.json | 48 +-
..._DIM_TABLE.csv => EDW.TEST_SELLER_TYPE_DIM.csv} | 0
examples/test_case_data/localmeta/kylin.properties | 3 +
.../test_case_data/parquet_test}/UUID | Bin
.../parquet_test/acl/legacy-test-domain-object | 31 +
.../bad_query/default.json | 0
.../parquet_test/cube/ci_inner_join_cube.json | 17 +
.../parquet_test/cube/ci_left_join_cube.json | 51 +
.../{localmeta => parquet_test}/cube/ssb.json | 0
.../parquet_test/cube_desc/ci_inner_join_cube.json | 381 +
.../parquet_test/cube_desc/ci_left_join_cube.json | 386 +
.../{localmeta => parquet_test}/cube_desc/ssb.json | 0
.../test_case_data/parquet_test/data/.gitignore | 1 +
.../data/DEFAULT.TEST_ACCOUNT.csv | 0
.../data/DEFAULT.TEST_CATEGORY_GROUPINGS.csv | 144 +
.../parquet_test/data/DEFAULT.TEST_COUNTRY.csv | 244 +
.../parquet_test/data/DEFAULT.TEST_KYLIN_FACT.csv | 10000 +++++++++++++++++++
.../parquet_test/data/DEFAULT.TEST_MEASURE.csv | 15 +
.../data/DEFAULT.TEST_ORDER.csv | 0
.../parquet_test/data/EDW.TEST_CAL_DT.csv | 731 ++
.../data/EDW.TEST_SELLER_TYPE_DIM.csv} | 0
.../data/EDW.TEST_SELLER_TYPE_DIM_TABLE.csv | 0
.../parquet_test/data/EDW.TEST_SITES.csv | 262 +
.../test_case_data/parquet_test/kylin.properties | 157 +
.../kylin_hive_conf.xml | 0
.../{localmeta => parquet_test}/kylin_job_conf.xml | 0
.../model_desc/ci_inner_join_model.json | 0
.../model_desc/ci_left_join_model.json | 0
.../model_desc/ssb.json | 0
.../parquet_test/project/default.json | 27 +
.../parquet_test/table/DEFAULT.TEST_ACCOUNT.json | 37 +
.../table/DEFAULT.TEST_CATEGORY_GROUPINGS.json | 156 +
.../parquet_test/table/DEFAULT.TEST_COUNTRY.json | 27 +
.../table/DEFAULT.TEST_KYLIN_FACT.json | 74 +
.../parquet_test/table/DEFAULT.TEST_ORDER.json | 36 +
.../parquet_test/table/EDW.TEST_CAL_DT.json | 414 +
.../table/EDW.TEST_SELLER_TYPE_DIM.json | 47 +
.../table/EDW.TEST_SELLER_TYPE_DIM_TABLE.json | 46 +
.../parquet_test/table/EDW.TEST_SITES.json | 52 +
.../parquet_test/table/SSB.CUSTOMER.json | 42 +
.../parquet_test/table/SSB.DATES.json | 78 +
.../parquet_test/table/SSB.PART.json | 46 +
.../parquet_test/table/SSB.SUPPLIER.json | 38 +
.../parquet_test/table/SSB.V_LINEORDER.json | 86 +
examples/test_case_data/sandbox/kylin.properties | 23 +-
jdbc/pom.xml | 2 +-
kylin-it/pom.xml | 2 +-
.../inmemcubing/ITDoggedCubeBuilderStressTest.java | 3 +-
.../cube/inmemcubing/ITDoggedCubeBuilderTest.java | 3 +-
.../cube/inmemcubing/ITInMemCubeBuilderTest.java | 5 +-
.../java/org/apache/kylin/query/KylinTestBase.java | 4 +-
.../apache/kylin/storage/hbase/ITStorageTest.java | 2 +-
.../test/resources/query/sql_function/query01.sql | 23 +
.../query/sql_intersect_count/query03.sql | 2 +-
.../query/sql_intersect_count/query04.sql | 33 +
.../query/sql_intersect_count/query05.sql | 23 +
.../{query32.sql => query32.sql.parquet.todo} | 0
.../{query33.sql => query33.sql.parquet.todo} | 0
.../{query35.sql => query35.sql.parquet.todo} | 0
.../test/resources/query/sql_subquery/query36.sql | 24 -
.../query/sql_subquery/query36.sql.parquet.todo | 24 +
.../test/resources/query/sql_tableau/query00.sql | 2 +
.../test/resources/query/sql_window/query00.sql | 2 +-
.../test/resources/query/sql_window/query09.sql | 4 +-
.../test/resources/query/sql_window/query12.sql | 7 +-
.../kylin-spark-classloader/pom.xml | 55 +
.../kylin/spark/classloader/ClassLoaderUtils.java | 78 +
.../spark/classloader/DebugTomcatClassLoader.java | 147 +
.../spark/classloader/KylinItClassLoader.java | 174 +
.../spark/classloader/KylinItSparkClassLoader.java | 189 +
.../kylin/spark/classloader/SparkClassLoader.java | 229 +
.../kylin/spark/classloader/TomcatClassLoader.java | 187 +
kylin-spark-project/kylin-spark-common/pom.xml | 55 +
.../spark/common/util/KapDateTimeUtils.scala | 128 +
.../engine/spark/common/util/TimeZoneUtils.java | 46 +
.../engine/spark/metadata/cube/PathManager.java | 42 +
.../apache/kylin/cluster/ClusterInfoFetcher.scala | 44 +
.../apache/kylin/query/implicits/implicits.scala | 40 +
.../main/scala/org/apache/spark/InfoHelper.scala | 40 +
.../src/main/scala/org/apache/spark/JobInfo.scala | 50 +
.../src/main/scala/org/apache/spark/Summary.scala | 215 +
.../org/apache/spark/dict/NBucketDictionary.java | 112 +
.../spark/dict/NGlobalDictBuilderAssist.scala | 77 +
.../apache/spark/dict/NGlobalDictHDFSStore.java | 301 +
.../org/apache/spark/dict/NGlobalDictMetaInfo.java | 98 +
.../org/apache/spark/dict/NGlobalDictStore.java | 72 +
.../org/apache/spark/dict/NGlobalDictionaryV2.java | 138 +
.../org/apache/spark/monitor/MemoryGetter.scala | 99 +
.../org/apache/spark/monitor/MemoryMonitor.scala | 130 +
.../spark/monitor/MetricsCollectHelper.scala | 54 +
.../org/apache/spark/monitor/MetricsSnapshot.scala | 23 +
.../apache/spark/monitor/MetricsThreadInfo.scala | 33 +
.../org/apache/spark/monitor/MonitorEnv.scala | 71 +
.../spark/monitor/MonitorExecutorExtension.scala | 60 +
.../org/apache/spark/monitor/MonitorManager.scala | 48 +
.../spark/monitor/MonitorMasterEndpoint.scala | 79 +
.../org/apache/spark/monitor/MonitorMessage.scala | 39 +
.../spark/monitor/MonitorSlaverEndpoint.scala | 79 +
.../scala/org/apache/spark/monitor/Reflector.scala | 86 +
.../spark/monitor/SparkMemoryManagerHandle.scala | 77 +
.../apache/spark/sql/KylinDataFrameManager.scala | 96 +
.../org/apache/spark/sql/KylinFunctions.scala | 159 +
.../sql/catalyst/expressions/DictEncodeImpl.scala | 56 +
.../sql/catalyst/expressions/ExpressionUtils.scala | 104 +
.../sql/catalyst/expressions/KapExpresssions.scala | 363 +
.../sql/catalyst/expressions/SparderAggFun.scala | 181 +
.../sql/catalyst/expressions/SplitPartImpl.scala | 32 +
.../catalyst/expressions/TimestampAddImpl.scala | 87 +
.../catalyst/expressions/TimestampDiffImpl.scala | 83 +
.../sql/catalyst/expressions/TruncateImpl.scala | 48 +
.../sql/execution/KylinFileSourceScanExec.scala | 268 +
.../spark/sql/execution/KylinJoinSelection.scala | 282 +
.../sql/execution/datasource/FilePruner.scala | 473 +
.../execution/datasource/KylinSourceStrategy.scala | 142 +
.../datasource/ResetShufflePartition.scala | 40 +
.../datasource/ShardFileStatusCache.scala | 33 +
.../apache/spark/sql/hive/utils/DetectItem.scala | 24 +
.../spark/sql/hive/utils/QueryMetricUtils.scala | 47 +
.../spark/sql/hive/utils/ResourceDetectUtils.scala | 167 +
.../spark/sql/udaf/ApproxCountDistinct.scala | 197 +
.../spark/sql/udaf/IntersectBitmapCounter.scala | 73 +
.../org/apache/spark/sql/udaf/IntersectCount.scala | 132 +
.../spark/sql/udaf/PreciseCountDistinct.scala | 174 +
.../org/apache/spark/utils/KylinReflectUtils.scala | 58 +
.../main/scala/org/apache/spark/utils/LogEx.scala | 61 +
.../apache/spark/sql/common/DebugFileSystem.scala | 133 +
.../apache/spark/sql/common/LocalMetadata.scala | 62 +
.../spark/sql/common/SharedSparkSession.scala | 134 +
.../spark/sql/common/SparderBaseFunSuite.scala | 82 +
.../apache/spark/sql/common/SparkQueryTest.scala | 172 +
kylin-spark-project/kylin-spark-engine/pom.xml | 199 +
.../kylin/engine/spark/NSparkCubingEngine.java | 61 +
.../spark/SparkBatchCubingEngineParquet.java | 65 +
.../engine/spark/application/SparkApplication.java | 262 +
.../kylin/engine/spark/application/SparkEntry.java | 27 +
.../engine/spark/builder/NBuildSourceInfo.java | 126 +
.../kylin/engine/spark/job/JobStepFactory.java | 72 +
.../apache/kylin/engine/spark/job/JobStepType.java | 27 +
.../engine/spark/job/NResourceDetectStep.java | 67 +
.../kylin/engine/spark/job/NSparkCubingJob.java | 141 +
.../kylin/engine/spark/job/NSparkCubingStep.java | 109 +
.../kylin/engine/spark/job/NSparkCubingUtil.java | 103 +
.../kylin/engine/spark/job/NSparkExecutable.java | 374 +
.../kylin/engine/spark/job/NSparkMergingJob.java | 116 +
.../kylin/engine/spark/job/NSparkMergingStep.java | 62 +
.../NSparkUpdateMetaAndCleanupAfterMergeStep.java | 74 +
.../merger/AfterMergeOrRefreshResourceMerger.java | 133 +
.../kylin/engine/spark/merger/MetadataMerger.java | 40 +
.../kylin/engine/spark/source/CsvSource.java | 131 +
.../spark/stats/utils/DateTimeCheckUtils.java | 80 +
.../kylin/engine/spark/utils/MetaDumpUtil.java | 130 +
.../engine/spark/utils/MockedDistributedLock.java | 159 +
.../kylin/engine/spark/utils/SchemaProcessor.scala | 60 +
.../kylin/cluster/ClusterInfoFetcherFactory.scala | 30 +
.../kylin/cluster/SchedulerInfoCmdHelper.scala | 138 +
.../org/apache/kylin/cluster/YarnInfoFetcher.scala | 54 +
.../cluster/parser/CapacitySchedulerParser.scala | 72 +
.../kylin/cluster/parser/FairSchedulerParser.scala | 64 +
.../kylin/cluster/parser/SchedulerParser.scala | 43 +
.../cluster/parser/SchedulerParserFactory.scala | 43 +
.../engine/spark/builder/CreateFlatTable.scala | 241 +
.../engine/spark/builder/DFBuilderHelper.scala | 89 +
.../engine/spark/builder/DFDictionaryBuilder.scala | 98 +
.../engine/spark/builder/DFLayoutMergeAssist.java | 95 +
.../engine/spark/builder/DFSnapshotBuilder.scala | 213 +
.../engine/spark/builder/DFTableEncoder.scala | 69 +
.../kylin/engine/spark/builder/DictHelper.scala | 40 +
.../spark/builder/DictionaryBuilderHelper.java | 148 +
.../engine/spark/cleanup/HDFSResourceCheck.scala | 31 +
.../kylin/engine/spark/job/BuildJobInfos.scala | 170 +
.../engine/spark/job/BuildLayoutWithUpdate.java | 119 +
.../kylin/engine/spark/job/CubeBuildJob.java | 362 +
.../kylin/engine/spark/job/CubeMergeJob.java | 198 +
.../kylin/engine/spark/job/CuboidAggregator.scala | 206 +
.../kylin/engine/spark/job/KylinBuildEnv.scala | 60 +
.../kylin/engine/spark/job/LogJobInfoUtils.scala | 89 +
.../kylin/engine/spark/job/MeasureEncoder.scala | 54 +
.../kylin/engine/spark/job/MeasureUDAF.scala | 186 +
.../engine/spark/job/ParentSourceChooser.scala | 210 +
.../spark/job/ResourceDetectBeforeCubingJob.java | 103 +
.../spark/job/ResourceDetectBeforeMergingJob.java | 97 +
.../kylin/engine/spark/job/SparkJobConstants.java | 27 +
.../apache/kylin/engine/spark/job/TopNUDAF.scala | 112 +
.../apache/kylin/engine/spark/job/UdfManager.scala | 103 +
.../engine/spark/scheduler/KylinJobEvent.scala | 44 +
.../engine/spark/scheduler/KylinJobListener.scala | 23 +
.../kylin/engine/spark/source/HiveSource.scala | 62 +
.../engine/spark/storage/ParquetDataStorage.java | 47 +
.../engine/spark/storage/ParquetStorage.scala | 34 +
.../kylin/engine/spark/utils/BuildUtils.scala | 108 +
.../kylin/engine/spark/utils/FileNames.scala | 28 +
.../kylin/engine/spark/utils/HDFSUtils.scala | 61 +
.../kylin/engine/spark/utils/JobMetrics.scala | 46 +
.../kylin/engine/spark/utils/JobMetricsUtils.scala | 141 +
.../kylin/engine/spark/utils/Repartitioner.java | 190 +
.../kylin/engine/spark/utils/SparkConfHelper.java | 110 +
.../engine/spark/utils/SparkConfRuleConstants.java | 27 +
.../kylin/engine/spark/utils/SparkDataSource.scala | 37 +
.../kylin/engine/spark/utils/SparkUtils.scala | 39 +
.../kylin/query/runtime/ExpressionConverter.scala | 346 +
.../util/UnsupportedSparkFunctionException.java | 40 +
.../org/apache/spark/application/JobMonitor.scala | 79 +
.../apache/spark/application/JobWorkSpace.scala | 110 +
.../org/apache/spark/application/JobWorker.scala | 67 +
.../spark/autoheal/ExceptionTerminator.scala | 123 +
.../org/apache/spark/conf/rule/SparkConfRule.scala | 168 +
.../apache/spark/scheduler/KylinJobEventLoop.scala | 48 +
.../scala/org/apache/spark/sql/KylinSparkEnv.scala | 121 +
.../org/apache/spark/sql/udf/SplitPartImpl.scala | 32 +
.../apache/spark/sql/udf/TimestampAddImpl.scala | 88 +
.../apache/spark/sql/udf/TimestampDiffImpl.scala | 83 +
.../org/apache/spark/sql/udf/TruncateImpl.scala | 48 +
.../org/apache/spark/util/KylinReflectUtils.scala | 58 +
.../org/apache/spark/utils/ProxyThreadUtils.scala | 39 +
.../org/apache/spark/utils/ResourceUtils.scala | 103 +
.../apache/spark/utils/YarnInfoFetcherUtils.java | 53 +
.../engine/spark/LocalWithSparkSessionTest.java | 295 +
.../apache/kylin/engine/spark/NSparkBasicTest.java | 66 +
.../engine/spark/dict/NGlobalDictionaryV2Test.java | 187 +
.../kylin/engine/spark/job/JobStepFactoryTest.java | 145 +
.../engine/spark/job/NSparkExecutableTest.java | 70 +
.../kylin/engine/spark/job/SparkCubingJobTest.java | 268 +
.../kylin/engine/spark/source/CsvSourceTest.java | 137 +
.../src/test/resources/data/ssb.csv | 10 +
.../src/test/resources/flattable.csv | 1 +
.../schedulerInfo/capacitySchedulerInfo.json | 1061 ++
.../resources/schedulerInfo/fairSchedulerInfo.json | 225 +
.../kylin-spark-engine/src/test/resources/ssb/UUID | Bin 0 -> 38 bytes
.../src/test/resources/ssb/cube/ssb.json | 15 +
.../src/test/resources/ssb/cube_desc/ssb.json | 239 +
.../src/test/resources/ssb/model_desc/ssb.json | 63 +
.../src/test/resources/ssb/project/ssb.json | 23 +
.../src/test/resources/ssb/table/SSB.CUSTOMER.json | 41 +
.../src/test/resources/ssb/table/SSB.DATES.json | 77 +
.../src/test/resources/ssb/table/SSB.PART.json | 45 +
.../test/resources/ssb/table/SSB.P_LINEORDER.json | 82 +
.../src/test/resources/ssb/table/SSB.SUPPLIER.json | 38 +
.../src/test/resources}/yarn-site.xml | 0
.../kylin/cluster/TestSchedulerInfoCmdHelper.scala | 46 +
.../scala/org/apache/kylin/cluster/TestUtils.scala | 37 +
.../parser/TestCapacitySchedulerParser.scala | 42 +
.../cluster/parser/TestFairSchedulerParser.scala | 48 +
.../parser/TestSchedulerParserFactory.scala | 36 +
.../engine/spark/builder/TestCreateFlatTable.scala | 140 +
.../engine/spark/builder/TestGlobalDictBuild.scala | 162 +
.../engine/spark/builder/TestSnapshotBuilder.scala | 211 +
.../kylin/engine/spark/job/TestCubeBuildJob.scala | 164 +
.../kylin/engine/spark/job/TestTopNUDAF.scala | 86 +
.../kylin/engine/spark/job/TestUdfManager.scala | 90 +
kylin-spark-project/kylin-spark-metadata/pom.xml | 71 +
.../kylin/engine/spark/metadata/cube/BitUtils.java | 47 +
.../engine/spark/metadata/cube/ManagerHub.java | 45 +
.../engine/spark/metadata/cube/StructField.java | 107 +
.../metadata/cube/model/ForestSpanningTree.java | 229 +
.../spark/metadata/cube/model/LayoutEntity.java | 170 +
.../spark/metadata/cube/model/SpanningTree.java | 79 +
.../spark/metadata/cube/source/NCubingEngine.java | 26 +
.../spark/metadata/cube/source/SourceFactory.java | 55 +
.../kylin/engine/spark/metadata/MetaData.scala | 132 +
.../engine/spark/metadata/MetadataConverter.scala | 256 +
.../spark/sql/utils/CuboidLayoutChooser.scala | 35 +
.../org/apache/spark/sql/utils/SparkTypeUtil.scala | 484 +
.../cube/model/NSpanningTreeCubingJobTest.java | 121 +
.../metadata/cube/model/NSpanningTreeTest.java | 185 +
kylin-spark-project/kylin-spark-query/pom.xml | 227 +
.../query/pushdown/PushDownRunnerSparkImpl.java | 71 +
.../kylin/query/pushdown/PushdownResponse.java | 49 +
.../kylin/query/pushdown/SparkSubmitter.java | 40 +
.../org/apache/kylin/query/SchemaProcessor.scala | 203 +
.../scala/org/apache/kylin/query/UdfManager.scala | 87 +
.../kylin/query/exception/SparderException.java | 25 +
.../query/exception/UnsupportedQueryException.java | 25 +
.../UnsupportedSparkFunctionException.java | 40 +
.../kylin/query/pushdown/SparkSqlClient.scala | 107 +
.../kylin/query/runtime/CalciteToSparkPlaner.scala | 124 +
.../kylin/query/runtime/DerivedProcess.scala | 225 +
.../kylin/query/runtime/ExpressionConverter.scala | 361 +
.../apache/kylin/query/runtime/RuntimeHelper.scala | 127 +
.../kylin/query/runtime/SparderLookupManager.scala | 86 +
.../kylin/query/runtime/SparderRexVisitor.scala | 339 +
.../apache/kylin/query/runtime/SparkEngine.java | 62 +
.../kylin/query/runtime/SparkOperation.scala | 56 +
.../kylin/query/runtime/plans/AggregatePlan.scala | 187 +
.../kylin/query/runtime/plans/FilterPlan.scala | 38 +
.../kylin/query/runtime/plans/JoinPlan.scala | 59 +
.../kylin/query/runtime/plans/LimitPlan.scala | 47 +
.../kylin/query/runtime/plans/ProjectPlan.scala | 70 +
.../kylin/query/runtime/plans/ResultPlan.scala | 191 +
.../kylin/query/runtime/plans/SortPlan.scala | 78 +
.../kylin/query/runtime/plans/TableScanPlan.scala | 285 +
.../kylin/query/runtime/plans/UnionPlan.scala | 45 +
.../kylin/query/runtime/plans/ValuesPlan.scala | 45 +
.../kylin/query/runtime/plans/WindowPlan.scala | 337 +
.../storage/spark/HadoopFileStorageQuery.java | 90 +
.../scala/org/apache/spark/sql/KylinSession.scala | 209 +
.../org/apache/spark/sql/SparderContext.scala | 301 +
.../sql/hive/KylinHiveSessionStateBuilder.scala | 55 +
.../pushdown/PushDownRunnerSparkImplTest.java | 158 +
kylin-spark-project/kylin-spark-test/pom.xml | 253 +
.../apache/kylin/engine/spark2/AccelerateInfo.java | 150 +
.../spark2/BuildAndQueryEmptySegmentsTest.java | 94 +
.../engine/spark2/NBadQueryAndPushDownTest.java | 136 +
.../apache/kylin/engine/spark2/NExecAndComp.java | 530 +
.../spark2/NManualBuildAndQueryCuboidTest.java | 202 +
.../engine/spark2/NManualBuildAndQueryTest.java | 341 +
.../engine/spark2/QueryInfoCollectorTest.java | 201 +
.../kylin/engine/spark2/QueryWithTableACLTest.java | 87 +
.../kylin/engine/spark2/auto/TestScenario.java | 115 +
.../spark2/file_pruning/NFilePruningTest.java | 388 +
.../kylin/engine/spark2/utils/QueryUtil.java | 159 +
.../spark2/utils/RecAndQueryCompareUtil.java | 162 +
.../test/resources/query/sql_pushdown/query11.sql | 9 +
.../metadata/_global/project/agg_match.json | 6 +
.../agg_match/table/DEFAULT.TEST_KYLIN_FACT.json | 74 +
kylin-spark-project/pom.xml | 200 +
metrics-reporter-hive/pom.xml | 11 +-
.../kylin/metrics/lib/impl/hive/HiveProducer.java | 59 +-
.../lib/impl/hive/HiveReservoirReporter.java | 6 +-
metrics-reporter-kafka/pom.xml | 2 +-
.../impl/kafka/KafkaActiveReserviorListener.java | 33 +-
.../lib/impl/kafka/KafkaReservoirReporter.java | 4 +-
parquet-assembly/pom.xml | 126 +
pom.xml | 201 +-
query/pom.xml | 2 +-
.../query/adhoc/JdbcPushDownConnectionManager.java | 23 +-
.../kylin/query/adhoc/PushDownRunnerJdbcImpl.java | 10 +-
.../kylin/query/enumerator/OLAPEnumerator.java | 5 +-
.../org/apache/kylin/query/exec/QueryEngine.java | 31 +
.../kylin/query/exec/QueryEngineFactory.java | 56 +
.../org/apache/kylin/query/exec/SparderMethod.java | 55 +
.../org/apache/kylin/query/exec/SparkExec.java | 75 +
.../apache/kylin/query/optrule/OLAPJoinRule.java | 23 +-
.../kylin/query/relnode/KylinAggregateCall.java | 4 +
.../kylin/query/relnode/OLAPAggregateRel.java | 2 +
.../apache/kylin/query/relnode/OLAPContext.java | 4 +-
.../apache/kylin/query/relnode/OLAPFilterRel.java | 2 +-
.../apache/kylin/query/relnode/OLAPJoinRel.java | 7 +
.../apache/kylin/query/relnode/OLAPProjectRel.java | 34 +-
.../apache/kylin/query/relnode/OLAPTableScan.java | 1 +
.../query/relnode/OLAPToEnumerableConverter.java | 49 +-
.../query/relnode/visitor/TupleFilterVisitor.java | 13 +-
.../apache/kylin/query/routing/QueryRouter.java | 5 +
.../apache/kylin/query/util/PushDownExecutor.java | 181 +
.../org/apache/kylin/query/util/PushDownUtil.java | 128 +-
.../org/apache/kylin/query/util/QueryUtil.java | 47 +-
.../apache/kylin/query/util/PushDownUtilTest.java | 7 +
.../org/apache/kylin/query/util/QueryUtilTest.java | 35 +
scalastyle_config.xml | 146 +
server-base/pom.xml | 45 +-
.../broadcaster/BroadcasterReceiveServlet.java | 24 +-
.../kylin/rest/controller/AdminController.java | 16 +
.../kylin/rest/controller/CubeController.java | 29 +-
.../kylin/rest/controller/DashboardController.java | 17 +-
.../rest/controller/ExternalFilterController.java | 12 +-
.../kylin/rest/controller/JobController.java | 37 +
.../kylin/rest/controller/MetricsController.java | 65 +
.../kylin/rest/controller/ModelController.java | 2 +
.../kylin/rest/controller/QueryController.java | 12 +-
.../ServiceDiscoveryStateController.java | 29 +-
.../kylin/rest/controller/StreamingController.java | 21 +-
.../rest/controller/StreamingV2Controller.java | 49 +-
.../kylin/rest/controller/TableController.java | 42 +
.../kylin/rest/controller/UserController.java | 7 +-
.../apache/kylin/rest/job/KylinHealthCheckJob.java | 48 +-
.../apache/kylin/rest/job/MetadataCleanupJob.java | 14 +-
.../apache/kylin/rest/job/StorageCleanupJob.java | 40 +-
.../kylin/rest/metrics/QueryMetricsFacade.java | 59 +-
.../java/org/apache/kylin/rest/msg/CnMessage.java | 4 +
.../java/org/apache/kylin/rest/msg/Message.java | 4 +
.../apache/kylin/rest/response/HBaseResponse.java | 9 +
.../kylin/rest/security/AclEntityFactory.java | 6 +
.../security/PasswordPlaceholderConfigurer.java | 2 +-
.../rest/security/cas/CasUserDetailsService.java | 80 +
.../saml/SAMLSimpleUserDetailsService.java | 64 +
.../org/apache/kylin/rest/service/CubeService.java | 125 +-
.../kylin/rest/service/DashboardService.java | 250 +-
.../kylin/rest/service/DiagnosisService.java | 5 +-
.../apache/kylin/rest/service/HBaseInfoUtil.java | 4 +-
.../org/apache/kylin/rest/service/JobService.java | 105 +-
.../kylin/rest/service/KylinUserService.java | 17 +-
.../apache/kylin/rest/service/MetricsService.java | 63 +
.../apache/kylin/rest/service/ModelService.java | 20 +-
.../apache/kylin/rest/service/ProjectService.java | 4 +-
.../apache/kylin/rest/service/QueryService.java | 12 +
.../kylin/rest/service/StreamingV2Service.java | 2 +-
.../apache/kylin/rest/service/TableService.java | 140 +-
.../org/apache/kylin/rest/util/ValidateUtil.java | 8 +-
.../kylin/rest/job/StorageCleanupJobTest.java | 2 +
server/pom.xml | 27 +-
.../java/org/apache/kylin/rest/DebugTomcat.java | 4 +
server/src/main/resources/applicationContext.xml | 10 +-
.../main/resources/kylin-security-cas-plugin.xml | 146 +
.../kylin-security-saml-noldap-plugin.xml | 305 +
server/src/main/resources/kylinSecurity.xml | 4 +
server/src/main/webapp/WEB-INF/web.xml | 1 +
.../kylin/rest/controller/CubeControllerTest.java | 14 +-
.../rest/controller/MetricsControllerTest.java | 41 +
.../kylin/rest/security/QueryWithTableACLTest.java | 89 -
.../kylin/rest/service/AdminServiceTest.java | 1 +
.../kylin/rest/service/KylinQueryTimeoutTest.java | 109 -
.../kylin/rest/service/QueryInfoCollectorTest.java | 200 -
source-hive/pom.xml | 6 +-
.../kylin/source/hive/BeelineHiveClient.java | 11 +-
.../apache/kylin/source/hive/CLIHiveClient.java | 34 +-
.../source/hive/CreateFlatHiveTableByLivyStep.java | 9 +
.../kylin/source/hive/CreateFlatHiveTableStep.java | 10 +-
.../kylin/source/hive/CreateMrHiveDictStep.java | 4 +-
.../apache/kylin/source/hive/HiveFlinkInput.java | 92 +
.../apache/kylin/source/hive/HiveInputBase.java | 80 +-
.../source/hive/HiveMetaStoreClientFactory.java | 62 +
.../org/apache/kylin/source/hive/HiveSource.java | 3 +
.../source/hive/RedistributeFlatHiveTableStep.java | 2 +-
.../java/org/apache/kylin/source/jdbc/CmdStep.java | 2 +-
source-jdbc/pom.xml | 2 +-
.../org/apache/kylin/source/jdbc/HiveCmdStep.java | 2 +-
.../org/apache/kylin/source/jdbc/JdbcExplorer.java | 26 +-
.../kylin/source/jdbc/JdbcHiveInputBase.java | 58 +-
.../apache/kylin/source/jdbc/JdbcHiveMRInput.java | 2 +-
.../kylin/source/jdbc/JdbcHiveSparkInput.java | 2 +-
.../org/apache/kylin/source/jdbc/JdbcSource.java | 6 +-
.../java/org/apache/kylin/source/jdbc/SqlUtil.java | 3 +-
.../kylin/source/jdbc/extensible/JdbcExplorer.java | 2 +
.../source/jdbc/extensible/JdbcHiveInputBase.java | 47 +-
.../source/jdbc/metadata/JdbcMetadataFactory.java | 4 +-
.../jdbc/metadata/PostgresqlJdbcMetadata.java | 68 +
.../jdbc/metadata/SQLServerJdbcMetadata.java | 2 +-
.../kylin/source/jdbc/sqoop/SqoopCmdStep.java | 2 +-
.../kylin/source/jdbc/JdbcHiveInputBaseTest.java | 2 +-
source-kafka/pom.xml | 2 +-
.../apache/kylin/source/kafka/KafkaInputBase.java | 6 +-
.../kafka/model/StreamCubeFactTableDesc.java | 3 +-
storage-hbase/pom.xml | 7 +-
.../kylin/storage/hbase/HBaseResourceStore.java | 5 +-
.../apache/kylin/storage/hbase/HBaseStorage.java | 4 +
.../hbase/cube/v2/CubeHBaseEndpointRPC.java | 9 +-
.../storage/hbase/cube/v2/CubeHBaseScanRPC.java | 4 +-
.../hbase/cube/v2/ExpectedSizeIterator.java | 3 +-
.../v2/coprocessor/endpoint/CubeVisitService.java | 19 +-
.../hbase/lookup/LookupTableToHFileJob.java | 4 +-
.../kylin/storage/hbase/steps/CreateHTableJob.java | 3 +-
.../hbase/steps/HBaseFlinkOutputTransition.java | 97 +
.../hbase/steps/HDFSPathGarbageCollectionStep.java | 1 -
.../storage/hbase/steps/HFileOutputFormat3.java | 3 +-
.../kylin/storage/hbase/util/CubeMigrationCLI.java | 61 +-
.../storage/hbase/util/DeployCoprocessorCLI.java | 7 +
.../kylin/storage/hbase/util/HBaseUnionUtil.java | 45 +
.../storage/hbase/util/StorageCleanupJob.java | 5 +-
storage-stream/pom.xml | 2 +-
.../kylin/storage/stream/StreamStorageQuery.java | 2 +-
stream-coordinator/pom.xml | 2 +-
.../kylin/stream/coordinator/Coordinator.java | 70 +-
.../stream/coordinator/StreamMetadataStore.java | 23 +
.../coordinator/ZookeeperStreamMetadataStore.java | 116 +-
.../stream/coordinator/assign/AssignmentUtil.java | 2 +-
.../coordinator/assign/AssignmentsCache.java | 66 +-
.../assign/CubePartitionRoundRobinAssigner.java | 4 +-
.../stream/coordinator/assign/DefaultAssigner.java | 9 +
.../coordinator/client/HttpCoordinatorClient.java | 2 +-
.../coordinator/coordinate/BuildJobSubmitter.java | 137 +-
.../coordinate/ReceiverClusterManager.java | 41 +-
.../coordinate/StreamingCoordinator.java | 19 +-
.../coordinator/doctor/ClusterStateChecker.java | 64 +-
.../kylin/stream/coordinator/CoordinatorTest.java | 5 +-
.../coordinate/BuildJobSubmitterTest.java | 23 +-
.../coordinator/coordinate/StreamingTestBase.java | 10 +-
stream-core/pom.xml | 6 +-
.../stream/core/client/ReceiverAdminClient.java | 5 +-
.../core/consumer/StreamingConsumerChannel.java | 34 +-
.../core/query/StreamingDataQueryPlanner.java | 21 +-
.../stream/core/query/StreamingTupleConverter.java | 4 +-
.../core/source/IStreamingMessageParser.java | 3 +
.../core/storage/StreamingSegmentManager.java | 2 +-
.../storage/columnar/ColumnarSegmentStore.java | 2 +-
.../ColumnarSegmentStoreFilesSearcher.java | 2 +-
.../storage/columnar/ColumnarStoreDimDesc.java | 5 +-
.../storage/columnar/ColumnarStoreMetricsDesc.java | 6 +-
.../columnar/FSInputGeneralColumnDataReader.java | 5 +-
.../storage/columnar/FragmentFileSearcher.java | 23 +-
.../core/storage/columnar/FragmentFilesMerger.java | 164 +-
.../storage/columnar/FragmentSearchResult.java | 18 +-
.../stream/core/storage/columnar/RawRecord.java | 26 +-
.../compress/FSInputLZ4CompressedColumnReader.java | 9 +-
.../compress/NoCompressedColumnReader.java | 4 +-
.../compress/RunLengthCompressedColumnReader.java | 6 +-
.../core/util/CompareFilterTimeRangeChecker.java | 17 +-
.../apache/kylin/stream/core/util/NodeUtil.java | 18 +-
.../stream/core/storage/CheckPointStoreTest.java | 11 +
.../core/storage/StreamingSegmentManagerTest.java | 113 +-
stream-receiver/pom.xml | 6 +-
.../stream/server/ReplicaSetLeaderSelector.java | 2 +-
.../kylin/stream/server/StreamingServer.java | 9 +
stream-source-kafka/pom.xml | 2 +-
.../stream/source/kafka/TimedJsonStreamParser.java | 41 +-
tomcat-ext/README.md | 5 -
tomcat-ext/pom.xml | 41 -
.../kylin/ext/CustomizedWebappClassloader.java | 67 -
tool-assembly/pom.xml | 17 +-
tool/pom.xml | 8 +-
.../apache/kylin/tool/AbstractInfoExtractor.java | 182 -
.../org/apache/kylin/tool/ClientEnvExtractor.java | 102 -
.../org/apache/kylin/tool/CubeMetaExtractor.java | 453 -
.../org/apache/kylin/tool/CubeMigrationCLI.java | 10 +-
.../org/apache/kylin/tool/DiagnosisInfoCLI.java | 55 +-
.../org/apache/kylin/tool/HBaseUsageExtractor.java | 223 -
.../java/org/apache/kylin/tool/HiveConfigCLI.java | 81 -
.../org/apache/kylin/tool/JobDiagnosisInfoCLI.java | 114 +-
.../apache/kylin/tool/JobInstanceExtractor.java | 223 -
.../org/apache/kylin/tool/KylinLogExtractor.java | 117 -
.../org/apache/kylin/tool/MrJobInfoExtractor.java | 322 -
.../kylin/tool/common/HadoopConfExtractor.java | 90 -
.../tool/extractor/AbstractInfoExtractor.java | 229 +
.../kylin/tool/extractor/ClientEnvExtractor.java | 140 +
.../kylin/tool/extractor/CubeMetaExtractor.java | 476 +
.../kylin/tool/extractor/HBaseUsageExtractor.java | 249 +
.../kylin/tool/extractor/HadoopConfExtractor.java | 92 +
.../apache/kylin/tool/extractor/HiveConfigCLI.java | 81 +
.../kylin/tool/extractor/JStackExtractor.java | 59 +
.../kylin/tool/extractor/JobInstanceExtractor.java | 224 +
.../kylin/tool/extractor/KylinLogExtractor.java | 113 +
.../kylin/tool/extractor/MrJobInfoExtractor.java | 324 +
.../tool/extractor/SparkEnvInfoExtractor.java | 91 +
.../kylin/tool/extractor/YarnLogExtractor.java | 194 +
.../tool/metrics/systemcube/CubeDescCreator.java | 42 +-
.../metrics/systemcube/CubeInstanceCreator.java | 26 +-
.../tool/metrics/systemcube/HiveTableCreator.java | 2 +-
.../tool/metrics/systemcube/KylinTableCreator.java | 34 +-
.../tool/metrics/systemcube/ModelCreator.java | 26 +-
.../kylin/tool/metrics/systemcube/SCCreator.java | 139 +-
.../metrics/systemcube/def/MetricsSinkDesc.java | 95 +
.../systemcube/streamingv2/KafkaTopicCreator.java | 46 +
.../streamingv2/StreamingMetadataCreator.java | 100 +
.../tool/metrics/systemcube/util/HiveSinkTool.java | 61 -
.../java/org/apache/kylin/tool/util/ToolUtil.java | 93 -
tool/src/main/resources/SCSinkTools.json | 33 +-
.../apache/kylin/tool/ClientEnvExtractorTest.java | 68 +
.../apache/kylin/tool/CubeMetaExtractorTest.java | 1 +
.../tool/metrics/systemcube/SCCreatorTest.java | 50 +-
tool/src/test/resources/SCSinkTools.json | 2 +-
webapp/Gruntfile.js | 68 +
webapp/app/index.html | 1 +
webapp/app/js/controllers/adminStreaming.js | 31 +
webapp/app/js/controllers/auth.js | 56 +-
webapp/app/js/controllers/cubeAdvanceSetting.js | 45 +-
webapp/app/js/controllers/cubeSchema.js | 5 +-
webapp/app/js/controllers/cubes.js | 2 +-
webapp/app/js/controllers/instances.js | 2 +-
webapp/app/js/controllers/job.js | 29 +
.../app/js/controllers/modelConditionsSettings.js | 26 +-
webapp/app/js/controllers/page.js | 26 +
webapp/app/js/controllers/query.js | 58 +-
webapp/app/js/controllers/sourceMeta.js | 674 +-
webapp/app/js/controllers/userProfile.js | 72 +
webapp/app/js/directives/directives.js | 13 +
webapp/app/js/directives/select.js | 4 +
webapp/app/js/model/cubeConfig.js | 6 +-
webapp/app/js/model/projectConfig.js | 2 +-
webapp/app/js/services/admin.js | 5 +-
webapp/app/js/services/jobs.js | 3 +-
webapp/app/js/services/kylinProperties.js | 28 +-
webapp/app/js/services/streaming.js | 11 +-
webapp/app/js/services/tables.js | 58 +-
webapp/app/less/app.less | 33 +-
webapp/app/less/component.less | 13 +
webapp/app/partials/admin/change_pwd.html | 14 +-
webapp/app/partials/admin/streaming.html | 3 +
webapp/app/partials/admin/user_create.html | 4 +-
.../partials/cubeDesigner/advanced_settings.html | 42 +-
webapp/app/partials/cubeDesigner/measures.html | 1 +
webapp/app/partials/cubes/cube_detail.html | 27 +-
webapp/app/partials/header.html | 28 +-
webapp/app/partials/jobs/jobList.html | 3 +
webapp/app/partials/jobs/job_steps.html | 2 +-
webapp/app/partials/login.html | 8 +
.../modelDesigner/conditions_settings.html | 63 +-
webapp/app/partials/models/models_tree.html | 2 +-
webapp/app/partials/query/query.html | 12 +
webapp/app/partials/query/query_detail.html | 2 +-
.../partials/tables/loadStreamingTableConfig.html | 11 +-
webapp/app/partials/tables/source_table_tree.html | 1 +
webapp/app/partials/tables/table_load.html | 66 +
webapp/app/partials/user/profile.html | 49 +
webapp/app/routes.json | 8 +
webapp/bower.json | 1 -
webapp/package.json | 3 +
881 files changed, 74184 insertions(+), 6190 deletions(-)
create mode 100644 .github/pull_request_template.md
create mode 100644 .github/workflows/maven.yml
create mode 100755 build/bin/download-flink.sh
create mode 100755 build/bin/find-flink-dependency.sh
create mode 100644 build/conf/kylin_job_conf_cube_merge.xml
create mode 100644 build/script/download-flink.sh
create mode 100644
core-common/src/main/java/org/apache/kylin/common/JobProcessContext.java
create mode 100644
core-common/src/main/java/org/apache/kylin/common/storage/DefaultStorageProvider.java
create mode 100644
core-common/src/main/java/org/apache/kylin/common/storage/IStorageProvider.java
create mode 100644
core-common/src/main/java/org/apache/kylin/common/util/FileUtils.java
create mode 100644
core-common/src/main/java/org/apache/kylin/common/util/FlinkEntry.java
create mode 100644
core-common/src/main/java/org/apache/kylin/common/util/ToolUtil.java
create mode 100644
core-common/src/main/java/org/apache/kylin/common/util/VersionUtil.java
create mode 100644
core-common/src/test/java/org/apache/kylin/common/util/CliCommandExecutorTest.java
create mode 100644
core-common/src/test/java/org/apache/kylin/common/util/TempMetadataBuilder.java
create mode 100644
core-common/src/test/java/org/apache/kylin/common/util/ToolUtilTest.java
create mode 100644
core-cube/src/main/java/org/apache/kylin/cube/common/TupleFilterNode.java
create mode 100644
core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java
create mode 100644
core-job/src/main/java/org/apache/kylin/job/exception/JobStoppedException.java
create mode 100644
core-job/src/main/java/org/apache/kylin/job/execution/CardinalityExecutable.java
create mode 100644
core-job/src/main/java/org/apache/kylin/job/execution/JobTypeEnum.java
create mode 100644
core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapIntersectValueAggFunc.java
create mode 100644
core-metadata/src/main/java/org/apache/kylin/measure/bitmap/RetentionPartialResult.java
create mode 100644
core-metadata/src/main/java/org/apache/kylin/metadata/model/CsvColumnDesc.java
create mode 100644
core-metadata/src/test/java/org/apache/kylin/metadata/model/PartitionDescTest.java
delete mode 100644
core-metrics/src/main/java/org/apache/kylin/metrics/lib/ActiveReservoirFilter.java
create mode 100644
core-metrics/src/main/java/org/apache/kylin/metrics/lib/ActiveReservoirRecordFilter.java
delete mode 100644
core-metrics/src/main/java/org/apache/kylin/metrics/lib/SinkTool.java
create mode 100644
datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/adaptor/PostgresqlAdaptor.java
delete mode 100644
datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/framework/conv/DefaultConfiguer.java
create mode 100644
datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/framework/conv/DefaultConfigurer.java
create mode 100644 datasource-sdk/src/main/resources/datasource/postgresql.xml
delete mode 100644
datasource-sdk/src/test/java/org/apache/kylin/sdk/datasource/framework/conv/GenericSqlConverterTest.java
create mode 100644
datasource-sdk/src/test/java/org/apache/kylin/sdk/datasource/framework/conv/PostgresqlSqlConverterTest.java
create mode 100644 docker/Dockerfile_dev
create mode 100644 docker/Dockerfile_hadoop
create mode 100644 docker/README.md
mode change 100644 => 100755 docker/run_container.sh
delete mode 100644 docker/use-guide.md
create mode 100644 engine-flink/pom.xml
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkBatchCubingEngine2.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkBatchCubingJobBuilder2.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkBatchMergeJobBuilder2.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkCubingByLayer.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkCubingMerge.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkExecutable.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkMergingDictionary.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkOnYarnConfigMapping.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/FlinkUtil.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/IFlinkInput.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/IFlinkOutput.java
create mode 100644
engine-flink/src/main/java/org/apache/kylin/engine/flink/util/PercentileCounterSerializer.java
create mode 100644
engine-flink/src/test/java/org/apache/kylin/engine/flink/FlinkOnYarnConfigMappingTest.java
create mode 100644
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeJobLockUtil.java
create mode 100644
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBuildDictionary.java
create mode 100644
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCreatingFlatTable.java
create mode 100644
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkSqlBatch.java
create mode 100644
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkUHCDictionary.java
create mode 100644
examples/test_case_data/file_prunning/cube/file_pruning_cube.json
create mode 100644
examples/test_case_data/file_prunning/cube/file_pruning_cube2.json
create mode 100644
examples/test_case_data/file_prunning/cube/file_pruning_cube_measure.json
create mode 100644
examples/test_case_data/file_prunning/cube_desc/file_pruning_cube.json
create mode 100644
examples/test_case_data/file_prunning/cube_desc/file_pruning_cube2.json
create mode 100644
examples/test_case_data/file_prunning/cube_desc/file_pruning_cube_measure.json
create mode 100644
examples/test_case_data/file_prunning/data/DEFAULT.TEST_KYLIN_FACT.csv
create mode 100644
examples/test_case_data/file_prunning/data/DEFAULT.TEST_MEASURE.csv
copy examples/test_case_data/{localmeta =>
file_prunning}/data/DEFAULT.TEST_ORDER.csv (100%)
create mode 100644 examples/test_case_data/file_prunning/kylin.properties
create mode 100644
examples/test_case_data/file_prunning/model_desc/file_pruning_model.json
create mode 100644
examples/test_case_data/file_prunning/model_desc/file_pruning_model2.json
create mode 100644
examples/test_case_data/file_prunning/model_desc/file_pruning_model_measure.json
create mode 100644 examples/test_case_data/file_prunning/project/default.json
create mode 100644
examples/test_case_data/file_prunning/table/DEFAULT.TEST_KYLIN_FACT.json
create mode 100644
examples/test_case_data/file_prunning/table/DEFAULT.TEST_MEASURE.json
create mode 100644
examples/test_case_data/file_prunning/table/DEFAULT.TEST_ORDER.json
copy
examples/test_case_data/localmeta/data/{EDW.TEST_SELLER_TYPE_DIM_TABLE.csv =>
EDW.TEST_SELLER_TYPE_DIM.csv} (100%)
copy {server-base/src/test/resources/test_meta =>
examples/test_case_data/parquet_test}/UUID (100%)
create mode 100644
examples/test_case_data/parquet_test/acl/legacy-test-domain-object
copy examples/test_case_data/{localmeta =>
parquet_test}/bad_query/default.json (100%)
create mode 100644
examples/test_case_data/parquet_test/cube/ci_inner_join_cube.json
create mode 100644
examples/test_case_data/parquet_test/cube/ci_left_join_cube.json
copy examples/test_case_data/{localmeta => parquet_test}/cube/ssb.json (100%)
create mode 100644
examples/test_case_data/parquet_test/cube_desc/ci_inner_join_cube.json
create mode 100644
examples/test_case_data/parquet_test/cube_desc/ci_left_join_cube.json
copy examples/test_case_data/{localmeta => parquet_test}/cube_desc/ssb.json
(100%)
create mode 100644 examples/test_case_data/parquet_test/data/.gitignore
copy examples/test_case_data/{localmeta =>
parquet_test}/data/DEFAULT.TEST_ACCOUNT.csv (100%)
create mode 100644
examples/test_case_data/parquet_test/data/DEFAULT.TEST_CATEGORY_GROUPINGS.csv
create mode 100644
examples/test_case_data/parquet_test/data/DEFAULT.TEST_COUNTRY.csv
create mode 100644
examples/test_case_data/parquet_test/data/DEFAULT.TEST_KYLIN_FACT.csv
create mode 100644
examples/test_case_data/parquet_test/data/DEFAULT.TEST_MEASURE.csv
copy examples/test_case_data/{localmeta =>
parquet_test}/data/DEFAULT.TEST_ORDER.csv (100%)
create mode 100644
examples/test_case_data/parquet_test/data/EDW.TEST_CAL_DT.csv
copy
examples/test_case_data/{localmeta/data/EDW.TEST_SELLER_TYPE_DIM_TABLE.csv =>
parquet_test/data/EDW.TEST_SELLER_TYPE_DIM.csv} (100%)
copy examples/test_case_data/{localmeta =>
parquet_test}/data/EDW.TEST_SELLER_TYPE_DIM_TABLE.csv (100%)
create mode 100644 examples/test_case_data/parquet_test/data/EDW.TEST_SITES.csv
create mode 100644 examples/test_case_data/parquet_test/kylin.properties
copy examples/test_case_data/{localmeta => parquet_test}/kylin_hive_conf.xml
(100%)
copy examples/test_case_data/{localmeta => parquet_test}/kylin_job_conf.xml
(100%)
copy {server-base/src/test/resources/test_meta =>
examples/test_case_data/parquet_test}/model_desc/ci_inner_join_model.json (100%)
copy examples/test_case_data/{localmeta =>
parquet_test}/model_desc/ci_left_join_model.json (100%)
copy examples/test_case_data/{localmeta => parquet_test}/model_desc/ssb.json
(100%)
create mode 100644 examples/test_case_data/parquet_test/project/default.json
create mode 100644
examples/test_case_data/parquet_test/table/DEFAULT.TEST_ACCOUNT.json
create mode 100644
examples/test_case_data/parquet_test/table/DEFAULT.TEST_CATEGORY_GROUPINGS.json
create mode 100644
examples/test_case_data/parquet_test/table/DEFAULT.TEST_COUNTRY.json
create mode 100644
examples/test_case_data/parquet_test/table/DEFAULT.TEST_KYLIN_FACT.json
create mode 100644
examples/test_case_data/parquet_test/table/DEFAULT.TEST_ORDER.json
create mode 100644
examples/test_case_data/parquet_test/table/EDW.TEST_CAL_DT.json
create mode 100644
examples/test_case_data/parquet_test/table/EDW.TEST_SELLER_TYPE_DIM.json
create mode 100644
examples/test_case_data/parquet_test/table/EDW.TEST_SELLER_TYPE_DIM_TABLE.json
create mode 100644
examples/test_case_data/parquet_test/table/EDW.TEST_SITES.json
create mode 100644 examples/test_case_data/parquet_test/table/SSB.CUSTOMER.json
create mode 100644 examples/test_case_data/parquet_test/table/SSB.DATES.json
create mode 100644 examples/test_case_data/parquet_test/table/SSB.PART.json
create mode 100644 examples/test_case_data/parquet_test/table/SSB.SUPPLIER.json
create mode 100644
examples/test_case_data/parquet_test/table/SSB.V_LINEORDER.json
create mode 100644 kylin-it/src/test/resources/query/sql_function/query01.sql
create mode 100644
kylin-it/src/test/resources/query/sql_intersect_count/query04.sql
create mode 100644
kylin-it/src/test/resources/query/sql_intersect_count/query05.sql
rename kylin-it/src/test/resources/query/sql_subquery/{query32.sql =>
query32.sql.parquet.todo} (100%)
rename kylin-it/src/test/resources/query/sql_subquery/{query33.sql =>
query33.sql.parquet.todo} (100%)
rename kylin-it/src/test/resources/query/sql_subquery/{query35.sql =>
query35.sql.parquet.todo} (100%)
delete mode 100644 kylin-it/src/test/resources/query/sql_subquery/query36.sql
create mode 100644
kylin-it/src/test/resources/query/sql_subquery/query36.sql.parquet.todo
create mode 100644 kylin-spark-project/kylin-spark-classloader/pom.xml
create mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/ClassLoaderUtils.java
create mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/DebugTomcatClassLoader.java
create mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/KylinItClassLoader.java
create mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/KylinItSparkClassLoader.java
create mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/SparkClassLoader.java
create mode 100644
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/TomcatClassLoader.java
create mode 100644 kylin-spark-project/kylin-spark-common/pom.xml
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/java/org/apache/kylin/engine/spark/common/util/KapDateTimeUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/java/org/apache/kylin/engine/spark/common/util/TimeZoneUtils.java
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/java/org/apache/kylin/engine/spark/metadata/cube/PathManager.java
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/kylin/cluster/ClusterInfoFetcher.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/kylin/query/implicits/implicits.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/InfoHelper.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/JobInfo.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/Summary.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/NBucketDictionary.java
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/NGlobalDictBuilderAssist.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/NGlobalDictHDFSStore.java
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/NGlobalDictMetaInfo.java
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/NGlobalDictStore.java
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/NGlobalDictionaryV2.java
create mode 100755
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MemoryGetter.scala
create mode 100755
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MemoryMonitor.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MetricsCollectHelper.scala
create mode 100755
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MetricsSnapshot.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MetricsThreadInfo.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MonitorEnv.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MonitorExecutorExtension.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MonitorManager.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MonitorMasterEndpoint.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MonitorMessage.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/MonitorSlaverEndpoint.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/Reflector.scala
create mode 100755
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/monitor/SparkMemoryManagerHandle.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/KylinDataFrameManager.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/KylinFunctions.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/DictEncodeImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExpressionUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/KapExpresssions.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/SparderAggFun.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/SplitPartImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimestampAddImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/TimestampDiffImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/TruncateImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/execution/KylinFileSourceScanExec.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/execution/KylinJoinSelection.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/execution/datasource/FilePruner.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/execution/datasource/KylinSourceStrategy.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/execution/datasource/ResetShufflePartition.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/execution/datasource/ShardFileStatusCache.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/hive/utils/DetectItem.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/hive/utils/QueryMetricUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/hive/utils/ResourceDetectUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/udaf/ApproxCountDistinct.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/udaf/IntersectBitmapCounter.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/udaf/IntersectCount.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/udaf/PreciseCountDistinct.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/utils/KylinReflectUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/utils/LogEx.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/test/java/org/apache/spark/sql/common/DebugFileSystem.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/test/java/org/apache/spark/sql/common/LocalMetadata.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/test/java/org/apache/spark/sql/common/SharedSparkSession.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/test/java/org/apache/spark/sql/common/SparderBaseFunSuite.scala
create mode 100644
kylin-spark-project/kylin-spark-common/src/test/java/org/apache/spark/sql/common/SparkQueryTest.scala
create mode 100644 kylin-spark-project/kylin-spark-engine/pom.xml
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/NSparkCubingEngine.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingEngineParquet.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/application/SparkApplication.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/application/SparkEntry.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/builder/NBuildSourceInfo.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/JobStepFactory.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/JobStepType.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NResourceDetectStep.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkCubingJob.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkCubingStep.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkCubingUtil.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkExecutable.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkMergingJob.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkMergingStep.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job/NSparkUpdateMetaAndCleanupAfterMergeStep.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/merger/AfterMergeOrRefreshResourceMerger.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/merger/MetadataMerger.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/source/CsvSource.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/stats/utils/DateTimeCheckUtils.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/utils/MetaDumpUtil.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/utils/MockedDistributedLock.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/utils/SchemaProcessor.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/ClusterInfoFetcherFactory.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/SchedulerInfoCmdHelper.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/YarnInfoFetcher.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/parser/CapacitySchedulerParser.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/parser/FairSchedulerParser.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/parser/SchedulerParser.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/cluster/parser/SchedulerParserFactory.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/CreateFlatTable.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DFBuilderHelper.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DFDictionaryBuilder.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DFLayoutMergeAssist.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DFSnapshotBuilder.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DFTableEncoder.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DictHelper.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/DictionaryBuilderHelper.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/cleanup/HDFSResourceCheck.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/BuildJobInfos.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/BuildLayoutWithUpdate.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/CubeBuildJob.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/CubeMergeJob.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/CuboidAggregator.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/KylinBuildEnv.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/LogJobInfoUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/MeasureEncoder.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/MeasureUDAF.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/ParentSourceChooser.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/ResourceDetectBeforeCubingJob.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/ResourceDetectBeforeMergingJob.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/SparkJobConstants.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/TopNUDAF.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/job/UdfManager.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/scheduler/KylinJobEvent.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/scheduler/KylinJobListener.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/source/HiveSource.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/storage/ParquetDataStorage.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/storage/ParquetStorage.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/BuildUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/FileNames.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/HDFSUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/JobMetrics.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/JobMetricsUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/Repartitioner.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/SparkConfHelper.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/SparkConfRuleConstants.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/SparkDataSource.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/utils/SparkUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/query/runtime/ExpressionConverter.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/query/util/UnsupportedSparkFunctionException.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/application/JobMonitor.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/application/JobWorkSpace.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/application/JobWorker.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/autoheal/ExceptionTerminator.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/conf/rule/SparkConfRule.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/scheduler/KylinJobEventLoop.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/sql/KylinSparkEnv.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/sql/udf/SplitPartImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/sql/udf/TimestampAddImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/sql/udf/TimestampDiffImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/sql/udf/TruncateImpl.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/util/KylinReflectUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/utils/ProxyThreadUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/utils/ResourceUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/utils/YarnInfoFetcherUtils.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/LocalWithSparkSessionTest.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/NSparkBasicTest.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/dict/NGlobalDictionaryV2Test.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/job/JobStepFactoryTest.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/job/NSparkExecutableTest.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/job/SparkCubingJobTest.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/source/CsvSourceTest.java
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/data/ssb.csv
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/flattable.csv
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/schedulerInfo/capacitySchedulerInfo.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/schedulerInfo/fairSchedulerInfo.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/UUID
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/cube/ssb.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/cube_desc/ssb.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/model_desc/ssb.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/project/ssb.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/table/SSB.CUSTOMER.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/table/SSB.DATES.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/table/SSB.PART.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/table/SSB.P_LINEORDER.json
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/ssb/table/SSB.SUPPLIER.json
copy {examples/test_case_data/sandbox =>
kylin-spark-project/kylin-spark-engine/src/test/resources}/yarn-site.xml (100%)
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/cluster/TestSchedulerInfoCmdHelper.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/cluster/TestUtils.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/cluster/parser/TestCapacitySchedulerParser.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/cluster/parser/TestFairSchedulerParser.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/cluster/parser/TestSchedulerParserFactory.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/builder/TestCreateFlatTable.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/builder/TestGlobalDictBuild.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/builder/TestSnapshotBuilder.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/job/TestCubeBuildJob.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/job/TestTopNUDAF.scala
create mode 100644
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/job/TestUdfManager.scala
create mode 100644 kylin-spark-project/kylin-spark-metadata/pom.xml
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/BitUtils.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/ManagerHub.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/StructField.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/model/ForestSpanningTree.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/model/LayoutEntity.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/model/SpanningTree.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/source/NCubingEngine.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/java/org/apache/kylin/engine/spark/metadata/cube/source/SourceFactory.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/scala/org/apache/kylin/engine/spark/metadata/MetaData.scala
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/scala/org/apache/kylin/engine/spark/metadata/MetadataConverter.scala
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/scala/org/apache/spark/sql/utils/CuboidLayoutChooser.scala
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/main/scala/org/apache/spark/sql/utils/SparkTypeUtil.scala
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/test/java/org/apache/kylin/engine/spark/metadata/cube/model/NSpanningTreeCubingJobTest.java
create mode 100644
kylin-spark-project/kylin-spark-metadata/src/test/java/org/apache/kylin/engine/spark/metadata/cube/model/NSpanningTreeTest.java
create mode 100644 kylin-spark-project/kylin-spark-query/pom.xml
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/java/org/apache/kylin/query/pushdown/PushDownRunnerSparkImpl.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/java/org/apache/kylin/query/pushdown/PushdownResponse.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/java/org/apache/kylin/query/pushdown/SparkSubmitter.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/SchemaProcessor.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/UdfManager.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/exception/SparderException.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/exception/UnsupportedQueryException.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/exception/UnsupportedSparkFunctionException.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/pushdown/SparkSqlClient.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/CalciteToSparkPlaner.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/DerivedProcess.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/ExpressionConverter.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/RuntimeHelper.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/SparderLookupManager.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/SparderRexVisitor.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/SparkEngine.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/SparkOperation.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/AggregatePlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/FilterPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/JoinPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/LimitPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/ProjectPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/ResultPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/SortPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/TableScanPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/UnionPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/ValuesPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/query/runtime/plans/WindowPlan.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/kylin/storage/spark/HadoopFileStorageQuery.java
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/spark/sql/KylinSession.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/spark/sql/SparderContext.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/main/scala/org/apache/spark/sql/hive/KylinHiveSessionStateBuilder.scala
create mode 100644
kylin-spark-project/kylin-spark-query/src/test/java/org/apache/kylin/query/pushdown/PushDownRunnerSparkImplTest.java
create mode 100644 kylin-spark-project/kylin-spark-test/pom.xml
create mode 100644
kylin-spark-project/kylin-spark-test/src/main/java/org/apache/kylin/engine/spark2/AccelerateInfo.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/BuildAndQueryEmptySegmentsTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/NBadQueryAndPushDownTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/NExecAndComp.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/NManualBuildAndQueryCuboidTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/NManualBuildAndQueryTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/QueryInfoCollectorTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/QueryWithTableACLTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/auto/TestScenario.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/file_pruning/NFilePruningTest.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/utils/QueryUtil.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/utils/RecAndQueryCompareUtil.java
create mode 100644
kylin-spark-project/kylin-spark-test/src/test/resources/query/sql_pushdown/query11.sql
create mode 100755
kylin-spark-project/kylin-spark-test/src/test/resources/ut_meta/agg_exact_match/metadata/_global/project/agg_match.json
create mode 100755
kylin-spark-project/kylin-spark-test/src/test/resources/ut_meta/agg_exact_match/metadata/agg_match/table/DEFAULT.TEST_KYLIN_FACT.json
create mode 100644 kylin-spark-project/pom.xml
create mode 100644 parquet-assembly/pom.xml
create mode 100644
query/src/main/java/org/apache/kylin/query/exec/QueryEngine.java
create mode 100644
query/src/main/java/org/apache/kylin/query/exec/QueryEngineFactory.java
create mode 100644
query/src/main/java/org/apache/kylin/query/exec/SparderMethod.java
create mode 100644
query/src/main/java/org/apache/kylin/query/exec/SparkExec.java
create mode 100644
query/src/main/java/org/apache/kylin/query/util/PushDownExecutor.java
create mode 100644 scalastyle_config.xml
create mode 100644
server-base/src/main/java/org/apache/kylin/rest/controller/MetricsController.java
create mode 100644
server-base/src/main/java/org/apache/kylin/rest/security/cas/CasUserDetailsService.java
create mode 100644
server-base/src/main/java/org/apache/kylin/rest/security/saml/SAMLSimpleUserDetailsService.java
create mode 100644
server-base/src/main/java/org/apache/kylin/rest/service/MetricsService.java
create mode 100644 server/src/main/resources/kylin-security-cas-plugin.xml
create mode 100644
server/src/main/resources/kylin-security-saml-noldap-plugin.xml
create mode 100644
server/src/test/java/org/apache/kylin/rest/controller/MetricsControllerTest.java
delete mode 100644
server/src/test/java/org/apache/kylin/rest/security/QueryWithTableACLTest.java
delete mode 100644
server/src/test/java/org/apache/kylin/rest/service/KylinQueryTimeoutTest.java
delete mode 100644
server/src/test/java/org/apache/kylin/rest/service/QueryInfoCollectorTest.java
create mode 100644
source-hive/src/main/java/org/apache/kylin/source/hive/HiveFlinkInput.java
create mode 100644
source-hive/src/main/java/org/apache/kylin/source/hive/HiveMetaStoreClientFactory.java
create mode 100644
source-jdbc/src/main/java/org/apache/kylin/source/jdbc/metadata/PostgresqlJdbcMetadata.java
create mode 100644
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseFlinkOutputTransition.java
create mode 100644
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/HBaseUnionUtil.java
delete mode 100644 tomcat-ext/README.md
delete mode 100644 tomcat-ext/pom.xml
delete mode 100644
tomcat-ext/src/main/java/org/apache/kylin/ext/CustomizedWebappClassloader.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/AbstractInfoExtractor.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/ClientEnvExtractor.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/CubeMetaExtractor.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/HBaseUsageExtractor.java
delete mode 100644 tool/src/main/java/org/apache/kylin/tool/HiveConfigCLI.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/JobInstanceExtractor.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/KylinLogExtractor.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/MrJobInfoExtractor.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/common/HadoopConfExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/AbstractInfoExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/ClientEnvExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/CubeMetaExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/HBaseUsageExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/HadoopConfExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/HiveConfigCLI.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/JStackExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/JobInstanceExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/KylinLogExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/MrJobInfoExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/SparkEnvInfoExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/extractor/YarnLogExtractor.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/def/MetricsSinkDesc.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/streamingv2/KafkaTopicCreator.java
create mode 100644
tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/streamingv2/StreamingMetadataCreator.java
delete mode 100644
tool/src/main/java/org/apache/kylin/tool/metrics/systemcube/util/HiveSinkTool.java
delete mode 100644 tool/src/main/java/org/apache/kylin/tool/util/ToolUtil.java
create mode 100644
tool/src/test/java/org/apache/kylin/tool/ClientEnvExtractorTest.java
create mode 100644 webapp/app/js/controllers/userProfile.js
create mode 100644 webapp/app/partials/user/profile.html
