[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072494#comment-13072494 ]
jirapos...@reviews.apache.org commented on HIVE-2272: ----------------------------------------------------- ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/ ----------------------------------------------------------- (Updated 2011-07-28 19:17:59.868991) Review request for hive. Changes ------- Decided to store timestamps as UTC on disk and add offsets (localize) if user needs through UDF. This simplifies use cases (compared to MySQL) where users from different time zones querying the same data would receive different results. Will file a follow up JIRA ticket to add table level timezone defaults. This would allow a user to import existing data that is already offset from UTC and write new data to the table with that offset. Summary: -Added config var for default timezone -Added GenericUDF for converting from UTC to localized timezone (either default, or supplied from list of Java timezones) Ex: utc_to_localized_timestamp(<timestamp>) OR utc_to_localized_timestamp(<timestamp>, "America/Chicago") -Fixed DateDiff bug I introduced Summary ------- Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. Supports SQL style jdbc timestamps of the format with nanosecond precision YYYY-MM-DD HH:MM:SS[.fff...] This addresses bug HIVE-2272. https://issues.apache.org/jira/browse/HIVE-2272 Diffs (updated) ----- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1151571 trunk/conf/hive-default.xml 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtcToLocalizedTimestamp.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_t_create3.q 1151571 trunk/ql/src/test/queries/clientpositive/timestamp_1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_comparison.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_udf.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_create_tbl1.q.out 1151571 trunk/ql/src/test/results/clientnegative/invalid_t_alter1.q.out 1151571 trunk/ql/src/test/results/clientnegative/invalid_t_alter2.q.out 1151571 trunk/ql/src/test/results/clientnegative/invalid_t_create1.q.out 1151571 trunk/ql/src/test/results/clientnegative/invalid_t_create2.q.out 1151571 trunk/ql/src/test/results/clientnegative/invalid_t_transform.q.out 1151571 trunk/ql/src/test/results/clientnegative/wrong_column_type.q.out 1151571 trunk/ql/src/test/results/clientpositive/show_functions.q.out 1151571 trunk/ql/src/test/results/clientpositive/timestamp_1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_comparison.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_udf.q.out PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestamp.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1151571 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1151571 Diff: https://reviews.apache.org/r/1135/diff Testing ------- Wrote some unit tests for string/binary serialization. Also ran against test data and looked consistent Thanks, Franklin > add TIMESTAMP data type > ----------------------- > > Key: HIVE-2272 > URL: https://issues.apache.org/jira/browse/HIVE-2272 > Project: Hive > Issue Type: New Feature > Reporter: Franklin Hu > Assignee: Franklin Hu > Attachments: hive-2272.1.patch, hive-2272.2.patch, hive-2272.3.patch, > hive-2272.4.patch, hive-2272.5.patch, hive-2272.6.patch, hive-2272.7.patch > > > Add TIMESTAMP type to serde2 that supports unix timestamp (1970-01-01 > 00:00:01 UTC to 2038-01-19 03:14:07 UTC) with optional nanosecond precision > using both LazyBinary and LazySimple SerDes. > For LazySimpleSerDe, the data is stored in jdbc compliant java.sql.Timestamp > parsable strings. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira