[jira] [Commented] (TRAFODION-2013) Hive null values for string datatype are not being handled correctly
[
https://issues.apache.org/jira/browse/TRAFODION-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303489#comment-15303489
]
Anoop Sharma commented on TRAFODION-2013:
-
During create of a hive delimited table, one can optionally specify
the string that represents a null value.
For ex:
create table t (a string) row format delimited fields terminated by ','
null defined as ';';
This will define the character semicolon (;) to be used to represent
a null value.
If the null string is defined as ''(two single quotes), then an empty
value will be used to represent null.
The specified null format string applies to all datatypes.
If 'null defined as' clause is not specified, then the default behavior
is similar to what was described earlier in this JIRA.
During read, the value '\N' will be treated as null for all datatypes, and
an empty string will be treated as null for non-varchar datatypes.
During writes, if a user specified 'null defined' clause doesn't exist,
then the value '\N' will be written for all datatypes.
> Hive null values for string datatype are not being handled correctly
>
>
> Key: TRAFODION-2013
> URL: https://issues.apache.org/jira/browse/TRAFODION-2013
> Project: Apache Trafodion
> Issue Type: Bug
>Reporter: Anoop Sharma
>Assignee: Anoop Sharma
>
> During reads, by default, a string of length zero in a hive file should be
> treated as a null value for non-string datatypes, and should be treated as
> string of length zero for string datatypes. A string of '\N' should be
> treated as null value for string datatypes.
> Similarly, during writes, a non-string datatype null should be written out
> as an empty string and a string datatype null value should be written out as
> '\N'.
> In Traf, during reads, a string of length zero is being returned as a null
> value for string datatypes. It is however, being correctly returned as a null
> value for non-string datatypes.
> And during writes, a null value is being written out as a string of length
> zero for string datatypes.
> examples of incorrect behavior:
> from hive:
> create table t (a string);
> from sqlci:
>insert into t values (null), ('a'), ('');
> from hive:
> hive> select * from t;
> OK
> t.a
> a
> Time taken: 0.358 seconds, Fetched: 3 row(s)
> Returns 3 rows with null row being written out as string of length zero.
> From sqlci:
> >>select * from hive.hive.t;
>A
>--
>?
>a
>?
>--- 3 row(s) selected.
> 2 rows with string of length zero are being returned as null rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2017) Tune HBase row cache sizes so UPDATE STATS completes
[ https://issues.apache.org/jira/browse/TRAFODION-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303376#comment-15303376 ] ASF GitHub Bot commented on TRAFODION-2017: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/505#discussion_r64850976 --- Diff: core/sql/ustat/hs_globals.cpp --- @@ -546,10 +557,13 @@ NABoolean HSGlobalsClass::setHBaseCacheSize(double sampleRatio) Int64 workableCacheSize = (Int64)(sampleRatio * calibrationFactor); if (workableCacheSize < 1) workableCacheSize = 1; // can't go below 1 unfortunately + else if (workableCacheSize > 50) +workableCacheSize = 50; - Int32 max = getDefaultAsLong(HBASE_NUM_CACHE_ROWS_MAX); - if ((workableCacheSize < 1) && // don't bother if 1 works - (max == 1)) // don't do it if user has already set this CQD + // if the user himself set the CQD, don't do anything + NADefaults &defs = ActiveSchemaDB()->getDefaults(); + if (defs.getProvenance(HBASE_NUM_CACHE_ROWS_MAX) == + NADefaults::INIT_DEFAULT_DEFAULTS) --- End diff -- Should we override it even if it is set in the default table. > Tune HBase row cache sizes so UPDATE STATS completes > > > Key: TRAFODION-2017 > URL: https://issues.apache.org/jira/browse/TRAFODION-2017 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: All, though prevalent on clusters with larger tables > and/or heavier loads >Reporter: David Wayne Birdsall >Assignee: David Wayne Birdsall > Fix For: 2.1-incubating > > > UPDATE STATISTICS often fails with HBase socket timeout exception and/or > scanner timeout exception when run with sampling on larger tables or clusters > with heavy concurrent workloads. > We have experimented in the past with setting various CQDs on large tables to > reduce these failures, however we were loathe to set them all the time due to > fears that this would lengthen elapsed time in non-failure scenarios. > Some recent work by Carol Pearson however shows that the increase in elapsed > time is negligible for smaller tables and in failure scenarios the failure > does not occur quickly, so paying a small penalty in elapsed time to increase > the probability of success seems a better trade-off. > Carol's work involves tables of less than 1 billion rows. The existing CQD > logic is still required for larger tables. But for tables of less than 1 > billion rows, she recommends setting HBASE_ROWS_CACHED_MIN and > HBASE_ROWS_CACHED_MAX to '50'. This JIRA is written to cover this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2017) Tune HBase row cache sizes so UPDATE STATS completes
[ https://issues.apache.org/jira/browse/TRAFODION-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303143#comment-15303143 ] ASF GitHub Bot commented on TRAFODION-2017: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/505#discussion_r64837288 --- Diff: core/sql/ustat/hs_globals.cpp --- @@ -546,10 +557,11 @@ NABoolean HSGlobalsClass::setHBaseCacheSize(double sampleRatio) Int64 workableCacheSize = (Int64)(sampleRatio * calibrationFactor); if (workableCacheSize < 1) workableCacheSize = 1; // can't go below 1 unfortunately + else if (workableCacheSize > 50) +workableCacheSize = 50; - Int32 max = getDefaultAsLong(HBASE_NUM_CACHE_ROWS_MAX); - if ((workableCacheSize < 1) && // don't bother if 1 works - (max == 1)) // don't do it if user has already set this CQD + Int32 maxDefault = getDefaultAsLong(HBASE_NUM_CACHE_ROWS_MAX); + if (maxDefault == 1) // don't do it if user has already set this CQD --- End diff -- Thanks for the suggestion for NADefaults::getProvenance(). I'll make this change. In the case that the user sets the CQD him/herself, I wanted to give them complete freedom to experiment as they please. Higher values may make UPDATE STATS run faster (though experiements suggest not by much). The trade-off in using higher values is an increased risk of failure. > Tune HBase row cache sizes so UPDATE STATS completes > > > Key: TRAFODION-2017 > URL: https://issues.apache.org/jira/browse/TRAFODION-2017 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: All, though prevalent on clusters with larger tables > and/or heavier loads >Reporter: David Wayne Birdsall >Assignee: David Wayne Birdsall > Fix For: 2.1-incubating > > > UPDATE STATISTICS often fails with HBase socket timeout exception and/or > scanner timeout exception when run with sampling on larger tables or clusters > with heavy concurrent workloads. > We have experimented in the past with setting various CQDs on large tables to > reduce these failures, however we were loathe to set them all the time due to > fears that this would lengthen elapsed time in non-failure scenarios. > Some recent work by Carol Pearson however shows that the increase in elapsed > time is negligible for smaller tables and in failure scenarios the failure > does not occur quickly, so paying a small penalty in elapsed time to increase > the probability of success seems a better trade-off. > Carol's work involves tables of less than 1 billion rows. The existing CQD > logic is still required for larger tables. But for tables of less than 1 > billion rows, she recommends setting HBASE_ROWS_CACHED_MIN and > HBASE_ROWS_CACHED_MAX to '50'. This JIRA is written to cover this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2004) Statistics on volatile tables is not supported
[
https://issues.apache.org/jira/browse/TRAFODION-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303095#comment-15303095
]
ASF GitHub Bot commented on TRAFODION-2004:
---
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/499#discussion_r64833315
--- Diff: core/sql/sqlcomp/nadefaults.cpp ---
@@ -6004,10 +6013,13 @@ enum DefaultConstants NADefaults::holdOrRestore
(const char *attrName,
char * value = NULL;
if (holdOrRestoreCQD == 1) // hold cqd
{
- if (heldDefaults_[attrEnum])
- {
- NADELETEBASIC(heldDefaults_[attrEnum], NADHEAP);
- }
+ if (heldHeldDefaults_[attrEnum])
+{
+ // Gasp! We've done three successive HOLDs... it's off to
+ // the bit bucket for the deepest value
+ NADELETEBASIC(heldHeldDefaults_[attrEnum], NADHEAP);
--- End diff --
My preference as a user would be an assert instead of incorrect behavior.
But this is probably a very little-used feature, so I doubt anyone will run
into this.
> Statistics on volatile tables is not supported
> --
>
> Key: TRAFODION-2004
> URL: https://issues.apache.org/jira/browse/TRAFODION-2004
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 2.0-incubating, 1.3-incubating
> Environment: All
>Reporter: David Wayne Birdsall
>Assignee: David Wayne Birdsall
>
> The Trafodion SQL Reference Manual on wiki:
> http://trafodion.apache.org/docs/sql_reference/Trafodion_SQL_Reference_Manual.pdf,
> Page 119, 'Considerations for CREATE VOLATILE TABLE' explicitly says the
> following:
> "Statistics are not automatically updated for volatile tables. If you need
> statistics, you must explicitly run UPDATE STATISTICS."
> However, as shown in the following example, update statistics does not work
> for volatile tables.
> >>create schema mytest;
> --- SQL operation complete.
> >>set schema mytest;
> --- SQL operation complete.
> >>
> >>create volatile table mytable (a int);
> --- SQL operation complete.
> >>insert into mytable values (1);
> --- 1 row(s) inserted.
> >>select * from mytable;
> A
> ---
> 1
> --- 1 row(s) selected.
> >>update statistics for table mytable on every column;
> *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is
> inaccessible.
> *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is
> inaccessible.
> *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is
> inaccessible.
> --- SQL operation failed with errors.
> >>
> >>drop volatile table mytable cascade;
> --- SQL operation complete.
> >>drop schema mytest cascade;
> --- SQL operation complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2017) Tune HBase row cache sizes so UPDATE STATS completes
[ https://issues.apache.org/jira/browse/TRAFODION-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303091#comment-15303091 ] ASF GitHub Bot commented on TRAFODION-2017: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/505#discussion_r64832927 --- Diff: core/sql/ustat/hs_globals.cpp --- @@ -546,10 +557,11 @@ NABoolean HSGlobalsClass::setHBaseCacheSize(double sampleRatio) Int64 workableCacheSize = (Int64)(sampleRatio * calibrationFactor); if (workableCacheSize < 1) workableCacheSize = 1; // can't go below 1 unfortunately + else if (workableCacheSize > 50) +workableCacheSize = 50; - Int32 max = getDefaultAsLong(HBASE_NUM_CACHE_ROWS_MAX); - if ((workableCacheSize < 1) && // don't bother if 1 works - (max == 1)) // don't do it if user has already set this CQD + Int32 maxDefault = getDefaultAsLong(HBASE_NUM_CACHE_ROWS_MAX); + if (maxDefault == 1) // don't do it if user has already set this CQD --- End diff -- This is existing code, but wanted to comment anyway: The code will break if we ever update HBASE_NUM_CACHE_ROWS_MAX in nadefaults.cpp and fail to update this file to the same value. There is another way to do this check: NADefaults::getProvenance() will return INIT_DEFAULT_DEFAULTS if a default hasn't been updated from the hard-coded defaults. I'm also wondering whether a higher value chosen by a user would be right here, given that we may use very small sample ratios. This may be different than the other workloads a user runs? Wouldn't it be better to use MIN(maxDefault, workableCacheSize)? > Tune HBase row cache sizes so UPDATE STATS completes > > > Key: TRAFODION-2017 > URL: https://issues.apache.org/jira/browse/TRAFODION-2017 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: All, though prevalent on clusters with larger tables > and/or heavier loads >Reporter: David Wayne Birdsall >Assignee: David Wayne Birdsall > Fix For: 2.1-incubating > > > UPDATE STATISTICS often fails with HBase socket timeout exception and/or > scanner timeout exception when run with sampling on larger tables or clusters > with heavy concurrent workloads. > We have experimented in the past with setting various CQDs on large tables to > reduce these failures, however we were loathe to set them all the time due to > fears that this would lengthen elapsed time in non-failure scenarios. > Some recent work by Carol Pearson however shows that the increase in elapsed > time is negligible for smaller tables and in failure scenarios the failure > does not occur quickly, so paying a small penalty in elapsed time to increase > the probability of success seems a better trade-off. > Carol's work involves tables of less than 1 billion rows. The existing CQD > logic is still required for larger tables. But for tables of less than 1 > billion rows, she recommends setting HBASE_ROWS_CACHED_MIN and > HBASE_ROWS_CACHED_MAX to '50'. This JIRA is written to cover this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2017) Tune HBase row cache sizes so UPDATE STATS completes
[ https://issues.apache.org/jira/browse/TRAFODION-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303043#comment-15303043 ] ASF GitHub Bot commented on TRAFODION-2017: --- GitHub user DaveBirdsall opened a pull request: https://github.com/apache/incubator-trafodion/pull/505 [TRAFODION-2017] Use a maximum HBase row cache of 50 for USTAT sample… … scans You can merge this pull request into a Git repository by running: $ git pull https://github.com/DaveBirdsall/incubator-trafodion Trafodion2017 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/505.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #505 commit d03526a2a508097dc8f23ac6fd63abb756a2a98e Author: Dave Birdsall Date: 2016-05-26T22:03:10Z [TRAFODION-2017] Use a maximum HBase row cache of 50 for USTAT sample scans > Tune HBase row cache sizes so UPDATE STATS completes > > > Key: TRAFODION-2017 > URL: https://issues.apache.org/jira/browse/TRAFODION-2017 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating > Environment: All, though prevalent on clusters with larger tables > and/or heavier loads >Reporter: David Wayne Birdsall >Assignee: David Wayne Birdsall > Fix For: 2.1-incubating > > > UPDATE STATISTICS often fails with HBase socket timeout exception and/or > scanner timeout exception when run with sampling on larger tables or clusters > with heavy concurrent workloads. > We have experimented in the past with setting various CQDs on large tables to > reduce these failures, however we were loathe to set them all the time due to > fears that this would lengthen elapsed time in non-failure scenarios. > Some recent work by Carol Pearson however shows that the increase in elapsed > time is negligible for smaller tables and in failure scenarios the failure > does not occur quickly, so paying a small penalty in elapsed time to increase > the probability of success seems a better trade-off. > Carol's work involves tables of less than 1 billion rows. The existing CQD > logic is still required for larger tables. But for tables of less than 1 > billion rows, she recommends setting HBASE_ROWS_CACHED_MIN and > HBASE_ROWS_CACHED_MAX to '50'. This JIRA is written to cover this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2015) hive/TEST018 fails in daily build regressions
[ https://issues.apache.org/jira/browse/TRAFODION-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303011#comment-15303011 ] ASF GitHub Bot commented on TRAFODION-2015: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/502 > hive/TEST018 fails in daily build regressions > -- > > Key: TRAFODION-2015 > URL: https://issues.apache.org/jira/browse/TRAFODION-2015 > Project: Apache Trafodion > Issue Type: Bug >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > It looks like CDH hive tests are failing both in Release 2.0 and master > because of java objects heap size is getting filled up. I suspect that > SequenceFileWriter objects are getting leaked. New tests added to could have > triggered the java objects heap memory to be filled up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (TRAFODION-1984) Installer packaging is not included in main package build target
[ https://issues.apache.org/jira/browse/TRAFODION-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuradha Hegde resolved TRAFODION-1984. --- Resolution: Fixed Fix Version/s: 2.1-incubating Fixed in PR 504 > Installer packaging is not included in main package build target > > > Key: TRAFODION-1984 > URL: https://issues.apache.org/jira/browse/TRAFODION-1984 > Project: Apache Trafodion > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 2.0-incubating >Reporter: Steve Varnau >Assignee: Anuradha Hegde > Fix For: 2.1-incubating > > > installer packaging is a separate build for some reason -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (TRAFODION-1274) LP Bug: 1465427 - DCSMaster web showing incorrect MXOSRVR process info
[ https://issues.apache.org/jira/browse/TRAFODION-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anuradha Hegde closed TRAFODION-1274. - Resolution: Cannot Reproduce not reproducible > LP Bug: 1465427 - DCSMaster web showing incorrect MXOSRVR process info > -- > > Key: TRAFODION-1274 > URL: https://issues.apache.org/jira/browse/TRAFODION-1274 > Project: Apache Trafodion > Issue Type: Bug > Components: connectivity-dcs >Reporter: JiepingZhang >Assignee: Anuradha Hegde > Fix For: 2.1-incubating > > > In the attached screen shot, column "Value" of row "Metrics" showed the > totalRegistered MXOSRVR is 114, totalAvailable MXOSRVR is 189. But actually > total MXOSRVR process on the Trafodion server is 120, among the 120 MXOSRVR > processes, 1 is in connected state, so the totalAvaiable MXOSRVR should be > 119. > Also, on the website, it shows 2 MXOSRVR are in CONNECTED state. Actually > these 2 MXOSRVR processes are the same process because they have the same > MXOSRVR process id: 19017. But 1 showed with the wrong "Start Time" -- "Tue > May 05 23:30:45 UTC 2015". I checked on the cluster, process 19107 was > started on June 07, not May 05. So the 2nd one is correct. > Below is the MXOSRVR process info on the Trafodion server side: > [trafodion@n015 scripts]$ sqps | grep mxosrvr|wc > 120 9609720 > [trafodion@n015 scripts]$ pdsh $MY_NODES ps -ef|grep 19017 > n015: 501 19017 18727 2 Jun07 ?04:33:34 mxosrvr -ZKHOST > n014.cm.cluster:2181,n015.cm.cluster:2181,n013.cm.cluster:2181 -RZ > g4q0091.houston.hp.com:3:1 -ZKPNODE /trafodion -CNGTO 60 -ZKSTO 180 -EADSCO 0 > -TCPADD 16.235.158.115 -MAXHEAPPCT 0 -STATISTICSINTERVAL 60 -STATISTICSLIMIT > 60 -STATISTICSTYPE aggregated -STATISTICSENABLE true -SQLPLAN true > -PORTMAPTOSECS -1 -PORTBINDTOSECS -1 > n015: 501 32642 21783 1 22:28 pts/700:00:00 grep 19017 > [trafodion@n015 scripts]$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1274) LP Bug: 1465427 - DCSMaster web showing incorrect MXOSRVR process info
[ https://issues.apache.org/jira/browse/TRAFODION-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302716#comment-15302716 ] Anuradha Hegde commented on TRAFODION-1274: --- Closing this bug as it is not re-producible. > LP Bug: 1465427 - DCSMaster web showing incorrect MXOSRVR process info > -- > > Key: TRAFODION-1274 > URL: https://issues.apache.org/jira/browse/TRAFODION-1274 > Project: Apache Trafodion > Issue Type: Bug > Components: connectivity-dcs >Reporter: JiepingZhang >Assignee: Anuradha Hegde > Fix For: 2.1-incubating > > > In the attached screen shot, column "Value" of row "Metrics" showed the > totalRegistered MXOSRVR is 114, totalAvailable MXOSRVR is 189. But actually > total MXOSRVR process on the Trafodion server is 120, among the 120 MXOSRVR > processes, 1 is in connected state, so the totalAvaiable MXOSRVR should be > 119. > Also, on the website, it shows 2 MXOSRVR are in CONNECTED state. Actually > these 2 MXOSRVR processes are the same process because they have the same > MXOSRVR process id: 19017. But 1 showed with the wrong "Start Time" -- "Tue > May 05 23:30:45 UTC 2015". I checked on the cluster, process 19107 was > started on June 07, not May 05. So the 2nd one is correct. > Below is the MXOSRVR process info on the Trafodion server side: > [trafodion@n015 scripts]$ sqps | grep mxosrvr|wc > 120 9609720 > [trafodion@n015 scripts]$ pdsh $MY_NODES ps -ef|grep 19017 > n015: 501 19017 18727 2 Jun07 ?04:33:34 mxosrvr -ZKHOST > n014.cm.cluster:2181,n015.cm.cluster:2181,n013.cm.cluster:2181 -RZ > g4q0091.houston.hp.com:3:1 -ZKPNODE /trafodion -CNGTO 60 -ZKSTO 180 -EADSCO 0 > -TCPADD 16.235.158.115 -MAXHEAPPCT 0 -STATISTICSINTERVAL 60 -STATISTICSLIMIT > 60 -STATISTICSTYPE aggregated -STATISTICSENABLE true -SQLPLAN true > -PORTMAPTOSECS -1 -PORTBINDTOSECS -1 > n015: 501 32642 21783 1 22:28 pts/700:00:00 grep 19017 > [trafodion@n015 scripts]$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (TRAFODION-1143) LP Bug: 1441712 - sqcheck shows negative value for mxosrvr count when there are stale mxosrvrs
[ https://issues.apache.org/jira/browse/TRAFODION-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-1143 started by Anuradha Hegde. - > LP Bug: 1441712 - sqcheck shows negative value for mxosrvr count when there > are stale mxosrvrs > -- > > Key: TRAFODION-1143 > URL: https://issues.apache.org/jira/browse/TRAFODION-1143 > Project: Apache Trafodion > Issue Type: Bug > Components: connectivity-dcs >Reporter: Aruna Sadashiva >Assignee: Anuradha Hegde > Fix For: 2.1-incubating > > > We have seen this several times. It seems to happen when there are leftover > mxosrvrs that did not get stopped by dcsstop. > [trafodion@n007 bin]$ sqcheck > Checking if processes are up. > Checking attempt: 1; user specified max: 2. Execution time in seconds: 0. > The SQ environment is up! > Process Configured Actual Down > --- -- -- > DTM 6 6 > RMS 12 12 > MXOSRVR 60 62 -2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TRAFODION-2017) Tune HBase row cache sizes so UPDATE STATS completes
David Wayne Birdsall created TRAFODION-2017: --- Summary: Tune HBase row cache sizes so UPDATE STATS completes Key: TRAFODION-2017 URL: https://issues.apache.org/jira/browse/TRAFODION-2017 Project: Apache Trafodion Issue Type: Bug Components: sql-cmp Affects Versions: 2.0-incubating Environment: All, though prevalent on clusters with larger tables and/or heavier loads Reporter: David Wayne Birdsall Assignee: David Wayne Birdsall Fix For: 2.1-incubating UPDATE STATISTICS often fails with HBase socket timeout exception and/or scanner timeout exception when run with sampling on larger tables or clusters with heavy concurrent workloads. We have experimented in the past with setting various CQDs on large tables to reduce these failures, however we were loathe to set them all the time due to fears that this would lengthen elapsed time in non-failure scenarios. Some recent work by Carol Pearson however shows that the increase in elapsed time is negligible for smaller tables and in failure scenarios the failure does not occur quickly, so paying a small penalty in elapsed time to increase the probability of success seems a better trade-off. Carol's work involves tables of less than 1 billion rows. The existing CQD logic is still required for larger tables. But for tables of less than 1 billion rows, she recommends setting HBASE_ROWS_CACHED_MIN and HBASE_ROWS_CACHED_MAX to '50'. This JIRA is written to cover this change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (TRAFODION-2001) Trafodion Elasticity enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on TRAFODION-2001 started by Gonzalo E Correa. --- > Trafodion Elasticity enhancements > - > > Key: TRAFODION-2001 > URL: https://issues.apache.org/jira/browse/TRAFODION-2001 > Project: Apache Trafodion > Issue Type: New Feature > Components: documentation, foundation, installer >Affects Versions: 2.1-incubating >Reporter: Gonzalo E Correa >Assignee: Gonzalo E Correa > Attachments: TRAFODION-2001-Elasticity.docx > > Original Estimate: 672h > Remaining Estimate: 672h > > Abstract > This JIRA proposes changes to the Apache Trafodion Incubation configuration > and operational elements used to define and operate a Trafodion cluster in a > Trafodion instance. The changes proposed build on existing functionality for > enhancing Trafodion to support elasticity by implementing the ability to add > and delete nodes used by Trafodion components. > The concept of a cluster is defined in Trafodion through the configuration of > nodes in the ‘sqconfig’ text file, the Trafodion configuration file. This > configuration file is compiled to generate a set of files used to start and > stop a Trafodion instance. The files generated consist of a configuration > database and scripts which presently contain fixed, i.e., hardcoded, > configuration topology attributes of a configured Trafodion instance. This > proposal addresses this hardcoded configuration topology issue by adding new > configuration commands in the Trafodion Foundation’s monitor components, > changing the methodology in the scripts generated, and extending the use of > the configuration database currently in use. > Objectives > The objective of this proposal is to implement elasticity capabilities in > Trafodion to match or exceed the elasticity capabilities of other modules in > the Hadoop eco system. > This focuses on the Trafodion Foundation functionality. Specifically, the > monitor components which use the configuration of nodes that define a > Trafodion cluster are enhanced to provide at set of primitive commands to add > and delete nodes as well as the generation of events to existing processes in > the instance of the addition or deletion of member nodes to the cluster > configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1977) Merge 2.0 fixes forward to master branch
[ https://issues.apache.org/jira/browse/TRAFODION-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302407#comment-15302407 ] Steve Varnau commented on TRAFODION-1977: - Completed, unless there are another round of 2.0.0 changes prior to official release. Leaving open for now. > Merge 2.0 fixes forward to master branch > > > Key: TRAFODION-1977 > URL: https://issues.apache.org/jira/browse/TRAFODION-1977 > Project: Apache Trafodion > Issue Type: Bug >Affects Versions: 2.1-incubating >Reporter: Steve Varnau >Assignee: Steve Varnau > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2004) Statistics on volatile tables is not supported
[ https://issues.apache.org/jira/browse/TRAFODION-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302287#comment-15302287 ] ASF GitHub Bot commented on TRAFODION-2004: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/499 > Statistics on volatile tables is not supported > -- > > Key: TRAFODION-2004 > URL: https://issues.apache.org/jira/browse/TRAFODION-2004 > Project: Apache Trafodion > Issue Type: Bug > Components: sql-cmp >Affects Versions: 2.0-incubating, 1.3-incubating > Environment: All >Reporter: David Wayne Birdsall >Assignee: David Wayne Birdsall > > The Trafodion SQL Reference Manual on wiki: > http://trafodion.apache.org/docs/sql_reference/Trafodion_SQL_Reference_Manual.pdf, > Page 119, 'Considerations for CREATE VOLATILE TABLE' explicitly says the > following: > "Statistics are not automatically updated for volatile tables. If you need > statistics, you must explicitly run UPDATE STATISTICS." > However, as shown in the following example, update statistics does not work > for volatile tables. > >>create schema mytest; > --- SQL operation complete. > >>set schema mytest; > --- SQL operation complete. > >> > >>create volatile table mytable (a int); > --- SQL operation complete. > >>insert into mytable values (1); > --- 1 row(s) inserted. > >>select * from mytable; > A > --- > 1 > --- 1 row(s) selected. > >>update statistics for table mytable on every column; > *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is > inaccessible. > *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is > inaccessible. > *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is > inaccessible. > --- SQL operation failed with errors. > >> > >>drop volatile table mytable cascade; > --- SQL operation complete. > >>drop schema mytest cascade; > --- SQL operation complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2004) Statistics on volatile tables is not supported
[
https://issues.apache.org/jira/browse/TRAFODION-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302271#comment-15302271
]
ASF GitHub Bot commented on TRAFODION-2004:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/499#discussion_r64769047
--- Diff: core/sql/sqlcomp/nadefaults.cpp ---
@@ -6004,10 +6013,13 @@ enum DefaultConstants NADefaults::holdOrRestore
(const char *attrName,
char * value = NULL;
if (holdOrRestoreCQD == 1) // hold cqd
{
- if (heldDefaults_[attrEnum])
- {
- NADELETEBASIC(heldDefaults_[attrEnum], NADHEAP);
- }
+ if (heldHeldDefaults_[attrEnum])
+{
+ // Gasp! We've done three successive HOLDs... it's off to
+ // the bit bucket for the deepest value
+ NADELETEBASIC(heldHeldDefaults_[attrEnum], NADHEAP);
--- End diff --
I considered this when developing the fix and decided against it. A person
can enter CQD HOLD commands at a sqlci prompt. With an assert, sqlci would
abend when a person did three such CQDs for the same default in succession.
Having user input cause an assert didn't seem good to me. Another possibility
is to raise an error or a warning. I will consider tihs for a later check-in.
> Statistics on volatile tables is not supported
> --
>
> Key: TRAFODION-2004
> URL: https://issues.apache.org/jira/browse/TRAFODION-2004
> Project: Apache Trafodion
> Issue Type: Bug
> Components: sql-cmp
>Affects Versions: 2.0-incubating, 1.3-incubating
> Environment: All
>Reporter: David Wayne Birdsall
>Assignee: David Wayne Birdsall
>
> The Trafodion SQL Reference Manual on wiki:
> http://trafodion.apache.org/docs/sql_reference/Trafodion_SQL_Reference_Manual.pdf,
> Page 119, 'Considerations for CREATE VOLATILE TABLE' explicitly says the
> following:
> "Statistics are not automatically updated for volatile tables. If you need
> statistics, you must explicitly run UPDATE STATISTICS."
> However, as shown in the following example, update statistics does not work
> for volatile tables.
> >>create schema mytest;
> --- SQL operation complete.
> >>set schema mytest;
> --- SQL operation complete.
> >>
> >>create volatile table mytable (a int);
> --- SQL operation complete.
> >>insert into mytable values (1);
> --- 1 row(s) inserted.
> >>select * from mytable;
> A
> ---
> 1
> --- 1 row(s) selected.
> >>update statistics for table mytable on every column;
> *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is
> inaccessible.
> *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is
> inaccessible.
> *** ERROR[4082] Object TRAFODION.MYTEST.MYTABLE does not exist or is
> inaccessible.
> --- SQL operation failed with errors.
> >>
> >>drop volatile table mytable cascade;
> --- SQL operation complete.
> >>drop schema mytest cascade;
> --- SQL operation complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2016) doesn't support vanilla hbase 1.1
[ https://issues.apache.org/jira/browse/TRAFODION-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15301923#comment-15301923 ] ASF GitHub Bot commented on TRAFODION-2016: --- GitHub user mashengchen opened a pull request: https://github.com/apache/incubator-trafodion/pull/503 TRAFODION-2016 change APACHE_1_0_X_SUPPORT to APACHE_1_1_X_SUPPORT. vanilla hbase 1.0 and 1.1 have several different, such as entend class, method return type.. so we can only support one version at one time. if users' hbase env is 1.0 set APACHE_1_0_X_SUPPORT="Y", not need to set APACHE_1_1_X_SUPPORT="N" if users' hbase env is 1.1 set APACHE_1_1_X_SUPPORT="Y", not need to set APACHE_1_0_X_SUPPORT="N" do not set APACHE_1_1_X_SUPPORT & APACHE_1_0_X_SUPPORT at the same time You can merge this pull request into a Git repository by running: $ git pull https://github.com/mashengchen/incubator-trafodion master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/503.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #503 commit 33e50e78226b1e6502e234f3d514a2f5801ca844 Author: root Date: 2016-05-25T07:15:22Z support hbase 1.1 commit b5219aff52c5a179ff1ab76112de504e6bdf62b9 Author: mashengchen Date: 2016-05-26T10:20:38Z Merge pull request #1 from apache/master update head > doesn't support vanilla hbase 1.1 > - > > Key: TRAFODION-2016 > URL: https://issues.apache.org/jira/browse/TRAFODION-2016 > Project: Apache Trafodion > Issue Type: Bug >Reporter: mashengchen >Assignee: mashengchen > > trafodion 2.1 support cdh5.4 & hdp2.3 & vanilla hbase 1.0. > but when use hbase 1.1 ,it can pass compile, the reason is some APIs and > method return type changed. > Need generate another suit of codes like what hdp2.3 do (hdp2.3 based on > hbase 1.1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TRAFODION-2016) doesn't support vanilla hbase 1.1
mashengchen created TRAFODION-2016: -- Summary: doesn't support vanilla hbase 1.1 Key: TRAFODION-2016 URL: https://issues.apache.org/jira/browse/TRAFODION-2016 Project: Apache Trafodion Issue Type: Bug Reporter: mashengchen Assignee: mashengchen trafodion 2.1 support cdh5.4 & hdp2.3 & vanilla hbase 1.0. but when use hbase 1.1 ,it can pass compile, the reason is some APIs and method return type changed. Need generate another suit of codes like what hdp2.3 do (hdp2.3 based on hbase 1.1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
