Partition Date Issue

[Debdutto Chakraborty]

Hi,

So, we have a hive table with analytical events data (impressions, clicks,
conversions and such). A typical day produces around 50 to 100 million rows
in this table with around 30 columns.

We were trying to move to Kylin and prepare cubes from the data which is in
this table.

Now the problem is:

   1. This hive table is partitioned on YEAR, MONTH, DAY columns. Which are
   separate columns.
   2. Kylin does not accept such separate columns as "Partition Date
   Column".
   3. Running Hive queries on non partitioned columns is a nightmare.


The only solution to this that I see is that give the user an option during
configuration to specify separate columns like this and then create the
query accordingly.

My only concern is that if this will impact the cube's "Refresh Settings"

Please let me know if this should be done. I'm open to do the development
and open a PR.

Regards,
Debdutto Chakraborty

Partition Date Issue

[Debdutto Chakraborty]

Hi,

So, we have a hive table with analytical events data (impressions, clicks,
conversions and such). A typical day produces around 50 to 100 million rows
in this table with around 30 columns.

We were trying to move to Kylin and prepare cubes from the data which is in
this table.

Now the problem is:

   1. This hive table is partitioned on YEAR, MONTH, DAY columns. Which are
   separate columns.
   2. Kylin does not accept such separate columns as "Partition Date
   Column".
   3. Running Hive queries on non partitioned columns is a nightmare.


The only solution to this that I see is that give the user an option during
configuration to specify separate columns like this and then create the
query accordingly.

My only concern is that if this will impact the cube's "Refresh Settings"

Please let me know if this should be done. I'm open to do the development
and open a PR.

Regards,
Debdutto Chakraborty

Build failed in Jenkins: Kylin-SonarQube #484

[Apache Jenkins Server]

See 

--
[...truncated 20.82 KB...]
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:report (post-test) @ kylin ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[PMD] No report found for mojo report
[INFO] 
[INFO] 
[INFO] Building Apache Kylin - External 2.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ kylin-external ---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (check-style) @ kylin-external ---
[INFO] Starting audit...
Audit done.
[CHECKSTYLE] Parsing file 

[CHECKSTYLE] Successfully parsed file 

 of module Apache Kylin - External with 0 unique warnings and 0 duplicates.
[CHECKSTYLE] Computing warning deltas based on reference build #483
[PMD] No report found for mojo check
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:prepare-agent (pre-test) @ kylin-external 
---
[INFO] surefireArgLine set to 
-javaagent:/home/jenkins/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ kylin-external 
---
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:exec (install_htrace) @ kylin-external ---
[INFO] Scanning for projects...
[INFO] 
[INFO] 
[INFO] Building Apache Kylin - External HTrace 2.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ kylin-external-htrace 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.17:check (check-style) @ 
kylin-external-htrace ---
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:prepare-agent (pre-test) @ 
kylin-external-htrace ---
[INFO] surefireArgLine set to 
-javaagent:/home/jenkins/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
kylin-external-htrace ---
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ 
kylin-external-htrace ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ 
kylin-external-htrace ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:copy (copy-jamm) @ 
kylin-external-htrace ---
[INFO] Configured Artifact: com.github.jbellis:jamm:?:jar
[INFO] Copying jamm-0.3.1.jar to 

[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ 
kylin-external-htrace ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:testCompile (default-testCompile) @ 
kylin-external-htrace ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ 
kylin-external-htrace ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.0:report (post-test) @ kylin-external-htrace 
---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO] 
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ kylin-external-htrace ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.5.1:attach-descriptor (attach-descriptor) @ 
kylin-external-htrace ---
[INFO] 
[INFO] --- maven-shade-plugin:3.0.0:shade (default) @ kylin-external-htrace ---
[INFO] Including org.apache.htrace:htrace-core:jar:3.1.0-incubating in the 
shaded jar.
[INFO] Including org.apache.htrace:htrace-zipkin:jar:3.1.0-incubating in the 
shaded jar.
[INFO] Including org.apache.thrift:libthrift:jar:0.9.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.2.5 in the shaded 
jar.

Jenkins build is back to normal : Kylin-SonarQube #485

[Apache Jenkins Server]

See 

[jira] [Created] (KYLIN-3382) YARN job link wasn't displayed when job is running

[Shaofeng SHI (JIRA)]

Shaofeng SHI created KYLIN-3382:
---

 Summary: YARN job link wasn't displayed when job is running
 Key: KYLIN-3382
 URL: https://issues.apache.org/jira/browse/KYLIN-3382
 Project: Kylin
  Issue Type: Bug
  Components: Spark Engine
Reporter: Shaofeng SHI


Affected steps: "Build cube with Spark"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3381) Stop/abort cubing job doesn't stop Spark job

[Shaofeng SHI (JIRA)]

Shaofeng SHI created KYLIN-3381:
---

 Summary: Stop/abort cubing job doesn't stop Spark job
 Key: KYLIN-3381
 URL: https://issues.apache.org/jira/browse/KYLIN-3381
 Project: Kylin
  Issue Type: Bug
  Components: Spark Engine
Reporter: Shaofeng SHI






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3380) Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive

[Dong Li (JIRA)]

Dong Li created KYLIN-3380:
--

 Summary: Allow to configure sqoop for jdbc source with a 
kylin_sqoop_conf.xml like hive
 Key: KYLIN-3380
 URL: https://issues.apache.org/jira/browse/KYLIN-3380
 Project: Kylin
  Issue Type: Improvement
  Components: RDBMS Source
Affects Versions: v2.3.1
Reporter: Dong Li


In hive source, users can config hive with a kylin_hive_conf.xml under conf 
directory. 
But for jdbc source, where we use sqoop to make flatten table, users cannot 
customize the configuration according to the real environment. To fix this, we 
can have a kylin_sqoop_conf.xml file under conf directory, then users can 
specify some configurations for sqoop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

timestampadd has some error

[jun]

kylin-1.6.0 version

1. SQL:  select part_dt, timestampadd(MONTH, 1, PART_DT) from KYLIN_SALES 
result: 2012-01-01, 2012-01-01
MONTH result not correct;

2. SQL: select part_dt, timestampadd(DAY, 1, PART_DT) from KYLIN_SALES 
result: 2012-01-01, 2012-01-02
DAY result corrcet

I found that DAY, HOUR, MINUTE correct, but MONTH and YEAR not work.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/