Thanks for your your suggestion,finally we changed the timestamp into date
format by sql and it worked.<br/>some other questions:<br/>1、what's the meaning
of 'shard by' column,is it proper to set partition column as the 'shard by'
column?<br/>2、Is there limitations on data format in kafka when building
streaming cubes？we succeed to build streaming a cube on the sample data
supplied by kylin,but failed on our own data,it's avro format,not json format.
在 2017-10-10 14:47:30，ShaoFeng Shi <[email protected]> 写道：
>Hi Miao,
>
>It doesn't understand your time format. You need to use the standard Date
>format in Hive. Or you can implement your own logic, with the interface "
>IPartitionConditionBuilder"
>
>2017-10-10 11:33 GMT+08:00 崔苗 <[email protected]>:
>
>> well,the timestamp column was bigint such as 1507547479434 in hive table,
>> when I define the endtime to build the cube ,I found the timestamp
>> 1507547479434 was converted to '20171009' and the log show that kylin
>> loaded data from hive with condition "WHERE (USER_REG.REG_TIME <
>> 20171009)",so the Intermediate Flat Hive Table was null. I want to know
>> could kylin derive other time values like “year_start”, “day_start” from
>> the bigint timestamp in hive as it does in kafka table? or we must change
>> the bigint timestamp into data format such as "2017-10-09" in hive?
>> At 2017-10-09 22:04:56, ShaoFeng Shi <[email protected]> wrote:
>> >Hi Miao,
>> >
>> >What's the error as you said: "kylin failed to load data from hive
>> tables"?
>> >
>> >In my opinion, it is not recommended to use timestamp as the partition
>> >column, since it is too fine granularity. Usually, the cube is partitioned
>> >by day/week/month; in some cases, it is by the hour; In streaming case, it
>> >might partition by the minute; But no case by timestamp. I put some
>> >comments about this in this document:
>> >https://kylin.apache.org/docs21/tutorial/cube_streaming.html
>> >
>> >2017-10-09 14:27 GMT+08:00 崔苗 <[email protected]>:
>> >
>> >> Hi,
>> >> we want to use tables in kafka as fact tables and tables in MySql as
>> >> lookup tables,so we put all the tables into hive and want to join them
>> as
>> >> cubes.
>> >>
>> >> the time column in fact table was timestamp, so does kylin2.1 support
>> >> timestamp for cube partition?
>> >> I find this :https://issues.apache.org/jira/browse/KYLIN-633 ,
>> >>
>> >> it seems kylin already supprt Timestamp for cube partition,but when we
>> >> define timestamp as partition , kylin failed to load data from hive
>> tables.
>> >>
>> >>
>> >> thanks in advanced for your reply.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >Best regards,
>> >
>> >Shaofeng Shi 史少锋
>>
>>
>>
>
>
>--
>Best regards,
>
>Shaofeng Shi 史少锋
