[jira] [Commented] (KYLIN-3335) Add project & cube related info to the job id for better filtering
[ https://issues.apache.org/jira/browse/KYLIN-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592364#comment-16592364 ] Zhong Yanghong commented on KYLIN-3335: --- Has not implemented yet? > Add project & cube related info to the job id for better filtering > -- > > Key: KYLIN-3335 > URL: https://issues.apache.org/jira/browse/KYLIN-3335 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Priority: Major > Fix For: Backlog > > > Currently it's painful to search cube or project related jobs, since those > infos are hidden in values. Especially, when users want to list job in a > period under one project, by current design, all of the job output info have > to be read into memory. If this kind of operation is done very often within a > short period, it's easy to get OOM. > If the job id is prefixed with project and cube names, then we can push down > prefix filters, which is efficient and safe. > This kind of change will cause backward compatibility issue. There're two > ways to deal with this: > * Set a milestone with time tag, if a search relates to data earlier than > this time, then just need to do one scan with prefix filter. Otherwise, two > scans are needed. One with prefix filter and the other use current strategy. > As time goes on, old job infos will be deleted. Once there's no data older > than the time, only one scan is needed. > * Do migration for the old data once. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3335) Add project & cube related info to the job id for better filtering
[ https://issues.apache.org/jira/browse/KYLIN-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556861#comment-16556861 ] Shaofeng SHI commented on KYLIN-3335: - [~yaho] any progress on this? > Add project & cube related info to the job id for better filtering > -- > > Key: KYLIN-3335 > URL: https://issues.apache.org/jira/browse/KYLIN-3335 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Priority: Major > Fix For: v2.5.0 > > > Currently it's painful to search cube or project related jobs, since those > infos are hidden in values. Especially, when users want to list job in a > period under one project, by current design, all of the job output info have > to be read into memory. If this kind of operation is done very often within a > short period, it's easy to get OOM. > If the job id is prefixed with project and cube names, then we can push down > prefix filters, which is efficient and safe. > This kind of change will cause backward compatibility issue. There're two > ways to deal with this: > * Set a milestone with time tag, if a search relates to data earlier than > this time, then just need to do one scan with prefix filter. Otherwise, two > scans are needed. One with prefix filter and the other use current strategy. > As time goes on, old job infos will be deleted. Once there's no data older > than the time, only one scan is needed. > * Do migration for the old data once. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3335) Add project & cube related info to the job id for better filtering
[ https://issues.apache.org/jira/browse/KYLIN-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458515#comment-16458515 ] liyang commented on KYLIN-3335: --- +1 for upgrade, prefer simpler approach :) > Add project & cube related info to the job id for better filtering > -- > > Key: KYLIN-3335 > URL: https://issues.apache.org/jira/browse/KYLIN-3335 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Priority: Major > > Currently it's painful to search cube or project related jobs, since those > infos are hidden in values. Especially, when users want to list job in a > period under one project, by current design, all of the job output info have > to be read into memory. If this kind of operation is done very often within a > short period, it's easy to get OOM. > If the job id is prefixed with project and cube names, then we can push down > prefix filters, which is efficient and safe. > This kind of change will cause backward compatibility issue. There're two > ways to deal with this: > * Set a milestone with time tag, if a search relates to data earlier than > this time, then just need to do one scan with prefix filter. Otherwise, two > scans are needed. One with prefix filter and the other use current strategy. > As time goes on, old job infos will be deleted. Once there's no data older > than the time, only one scan is needed. > * Do migration for the old data once. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3335) Add project & cube related info to the job id for better filtering
[ https://issues.apache.org/jira/browse/KYLIN-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431825#comment-16431825 ] Zhong Yanghong commented on KYLIN-3335: --- For me, I prefer the simple way by do migration once. [~liyang.g...@gmail.com] and [~Shaofengshi], what do you think? > Add project & cube related info to the job id for better filtering > -- > > Key: KYLIN-3335 > URL: https://issues.apache.org/jira/browse/KYLIN-3335 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Priority: Major > > Currently it's painful to search cube or project related jobs, since those > infos are hidden in values. Especially, when users want to list job in a > period under one project, by current design, all of the job output info have > to be read into memory. If this kind of operation is very often with a short > period, it's easy to get OOM. > If the job id is prefixed with project and cube names, then we can push down > prefix filters, which is efficient and safe. > This kind of change will cause backward compatibility issue. There're two > ways to deal with this: > * Set a milestone with time tag, if a search relates to data earlier than > this time, then just need to do one scan with prefix filter. Otherwise, two > scans are needed. One with prefix filter and the other use current strategy. > As time goes on, old job infos will be deleted. Once there's no data older > than the time, only one scan is needed. > * Do migration for the old data once. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
