+1
On Fri, 15 Dec 2017 at 5:34 PM, Mohammad Shahid Khan <
mohdshahidkhan1...@gmail.com> wrote:
> Hi All,
> Please find the details.
> Please response as earliest as possible.
> *Problem*:
>
> if we move the table to environment having different timezone or we change
> the system current timezone
> then after IUD operation some of the blocks are not treated as valid
> blocks.
>
> [{"timestamp":"15-12-2017
>
> 16:50:31:703","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",*"loadStartTime":"15-12-2017
> 16:50:27:493"*,"visibility":"true","fileFormat":"COLUMNAR_V3"}]
>
> part-0-0_batchno0-0-*1513336827493*.carbondata
>
> If timezone is different than at the time load was done, the value
> calculated from *loadStartTime *15-12-2017 16:50:27:493 will not match to
> the time stamp extracted from block file name.
>
> *Solution*:
>
> We should stop writing the loadStartTime and timestamp in "dd-MM-yyyy
> HH:mm:ss:SSS" format.
> *We should write the long value of the timestamp.*
> *like below:*
> [{"timestamp":"*1513336827593*
>
> ","loadStatus":"Success","loadName":"0","partitionCount":"0","isDeleted":"FALSE","dataSize":"912","indexSize":"700","updateDeltaEndTimestamp":"","updateDeltaStartTimestamp":"","updateStatusFileName":"",
> *"loadStartTime":"**1513336827493**"*,
> "visibility":"true","fileFormat":"COLUMNAR_V3"}]
>
> Backward compatibility:
> If string to Long parse fail, we can fall back for date parsing.
>
> private long convertTimeStampToLong(String factTimeStamp) {
> try {
> // for new loads the factTimeStamp will be long string
> // but for the old store the it will be in form of date string
> *return Long.parseLong(factTimeStamp);*
> } catch (NumberFormatException nf) {
> SimpleDateFormat parser = new
> SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP_MILLIS);
> Date dateToStr = null;
> try {
> dateToStr = parser.parse(factTimeStamp);
> return dateToStr.getTime();
> } catch (ParseException e) {
> LOGGER
> .error("Cannot convert" + factTimeStamp + " to Time/Long type
> value" + e.getMessage());
> parser = new
> SimpleDateFormat(CarbonCommonConstants.CARBON_TIMESTAMP);
> try {
> // if the load is in progress, factTimeStamp will be null, so use
> current time
> if (null == factTimeStamp) {
> return System.currentTimeMillis();
> }
> dateToStr = parser.parse(factTimeStamp);
> return dateToStr.getTime();
> } catch (ParseException e1) {
> LOGGER.error(
> "Cannot convert" + factTimeStamp + " to Time/Long type value"
> + e1.getMessage());
> return 0;
> }
> }
> }
> }
>
>
> Regards,
> Mohammad Shahid Khan
>
