[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[ https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187629#comment-15187629 ] ASF GitHub Bot commented on TRAFODION-1848: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/363 > LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr > process > -- > > Key: TRAFODION-1848 > URL: https://issues.apache.org/jira/browse/TRAFODION-1848 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-exe >Affects Versions: any >Reporter: Sandhya Sundaresan >Assignee: Sandhya Sundaresan > Fix For: 2.0-incubating > > Original Estimate: 672h > Remaining Estimate: 672h > > Need support to compact lob data files after a certain threshold is reached . > Deleting or updating a table with lob columns cn cause the lob datafiles to > be left with unused holes. These need to be compacted regularly. > TO simplify the flow of execution, the mxlobsrvr process functionality is > being moved into the master executor. There is no need for a spearate process > anymore since there isn't going to be a separate storage engine that may need > to make calls to this process. The lob file processing can all be done in > the master process itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186177#comment-15186177
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user sandhyasun commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55454391
--- Diff: core/sql/cli/Cli.cpp ---
@@ -9684,15 +9640,17 @@ Lng32 SQLCLI_LOBcliInterface
cliRC = cliInterface->executeImmediate(query);
currContext.resetSqlParserFlags(0x1);
-
+
if (cliRC < 0)
{
cliInterface->retrieveSQLDiagnostics(myDiags);
-
+ if (myDiags->containsError(-CAT_OBJECT_DOES_NOT_EXIST_IN_TRAFODION))
--- End diff --
Found a related issue with drop schema and ddl transactions. I am going to
backout this change and not filter out any errors. It does not help with ddl
transactions - the diagnostics not being cleared causes problems during
transaction commit. So opened JIRA TRAFODION-1883 and will fix it shortly (as
part of a separate pull request) .
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185727#comment-15185727
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55421989
--- Diff: core/sql/cli/Cli.cpp ---
@@ -9684,15 +9640,17 @@ Lng32 SQLCLI_LOBcliInterface
cliRC = cliInterface->executeImmediate(query);
currContext.resetSqlParserFlags(0x1);
-
+
if (cliRC < 0)
{
cliInterface->retrieveSQLDiagnostics(myDiags);
-
+ if (myDiags->containsError(-CAT_OBJECT_DOES_NOT_EXIST_IN_TRAFODION))
--- End diff --
Right, but myDiags could contain more than one error, right? What happens
then?
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185676#comment-15185676
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user sandhyasun commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55418965
--- Diff: core/sql/exp/ExpLOB.cpp ---
@@ -186,13 +173,114 @@ Lng32 ExpLOBoper::createLOB(void * lobGlob, void *
lobHeap,
else
lobGlobL = lobGlob;
- rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,(char *)"default",lobMaxSize);
+ rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,hdfsServer,lobMaxSize, hdfsPort);
+
+ return rc;
+}
+void ExpLOBoper::calculateNewOffsets(ExLobInMemoryDescChunksEntry
*dcArray, Lng32 numEntries)
+{
+ Int32 i = 0;
+ //Check if there is a hole right up front for the first entry. If so
start compacting with the first entry.
+ if (dcArray[0].getCurrentOffset() != 0)
+{
+ dcArray[0].setNewOffset(0);
+ for (i = 1; i < numEntries; i++)
+{
+ dcArray[i].setNewOffset(dcArray[i-1].getNewOffset() +
dcArray[i-1].getChunkLen());
+}
+}
+ else
+//Look for the first unused section and start compacting from there.
+{
+ NABoolean done = FALSE;
+ i = 0;
+ Int32 j = 0;
+ while (i < numEntries && !done )
+{
+ if ((dcArray[i].getCurrentOffset()+dcArray[i].getChunkLen()) !=
+ dcArray[i+1].getCurrentOffset())
+{
+ j = i+1;
+ while (j < numEntries)
+{
+
dcArray[j].setNewOffset(dcArray[j-1].getNewOffset()+dcArray[j-1].getChunkLen());
+ j++;
+}
+ done = TRUE;
+}
+ i++;
+}
+}
+ return ;
+}
+
+Lng32 ExpLOBoper::compactLobDataFile(void
*lobGlob,ExLobInMemoryDescChunksEntry *dcArray,Int32 numEntries,char
*tgtLobName,Int64 lobMaxChunkMemSize, void *lobHeap, char *hdfsServer, Int32
hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ // Call ExeLOBinterface to create the LOB
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+
+ if (rc)
+ return -1;
+
+ rc = ExpLOBinterfacePerformGC(lobGlobL,tgtLobName, (void *)dcArray,
numEntries,hdfsServer,hdfsPort,lobLoc,lobMaxChunkMemSize);
+
+ return rc;
+}
+
+Int32 ExpLOBoper::restoreLobDataFile(void *lobGlob, char *lobName, void
*lobHeap, char *hdfsServer, Int32 hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+ rc = initLOBglobal(lobGlobL, lobHeap);
--- End diff --
You found a defect. The if is needed and we cannot call initLOBglobal if
lobGlob was passed in. Correcting that.
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185665#comment-15185665
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user sandhyasun commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55418461
--- Diff: core/sql/cli/Cli.cpp ---
@@ -9684,15 +9640,17 @@ Lng32 SQLCLI_LOBcliInterface
cliRC = cliInterface->executeImmediate(query);
currContext.resetSqlParserFlags(0x1);
-
+
if (cliRC < 0)
{
cliInterface->retrieveSQLDiagnostics(myDiags);
-
+ if (myDiags->containsError(-CAT_OBJECT_DOES_NOT_EXIST_IN_TRAFODION))
--- End diff --
We will ignore only this particular error. If it's any other error, we go
to "error_return". That will return an error to the calling layer.
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185300#comment-15185300
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55393471
--- Diff: core/sql/exp/ExpLOB.cpp ---
@@ -186,13 +173,114 @@ Lng32 ExpLOBoper::createLOB(void * lobGlob, void *
lobHeap,
else
lobGlobL = lobGlob;
- rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,(char *)"default",lobMaxSize);
+ rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,hdfsServer,lobMaxSize, hdfsPort);
+
+ return rc;
+}
+void ExpLOBoper::calculateNewOffsets(ExLobInMemoryDescChunksEntry
*dcArray, Lng32 numEntries)
+{
+ Int32 i = 0;
+ //Check if there is a hole right up front for the first entry. If so
start compacting with the first entry.
+ if (dcArray[0].getCurrentOffset() != 0)
+{
+ dcArray[0].setNewOffset(0);
+ for (i = 1; i < numEntries; i++)
+{
+ dcArray[i].setNewOffset(dcArray[i-1].getNewOffset() +
dcArray[i-1].getChunkLen());
+}
+}
+ else
+//Look for the first unused section and start compacting from there.
+{
+ NABoolean done = FALSE;
+ i = 0;
+ Int32 j = 0;
+ while (i < numEntries && !done )
+{
+ if ((dcArray[i].getCurrentOffset()+dcArray[i].getChunkLen()) !=
+ dcArray[i+1].getCurrentOffset())
+{
+ j = i+1;
+ while (j < numEntries)
+{
+
dcArray[j].setNewOffset(dcArray[j-1].getNewOffset()+dcArray[j-1].getChunkLen());
+ j++;
+}
+ done = TRUE;
+}
+ i++;
+}
+}
+ return ;
+}
+
+Lng32 ExpLOBoper::compactLobDataFile(void
*lobGlob,ExLobInMemoryDescChunksEntry *dcArray,Int32 numEntries,char
*tgtLobName,Int64 lobMaxChunkMemSize, void *lobHeap, char *hdfsServer, Int32
hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ // Call ExeLOBinterface to create the LOB
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+
+ if (rc)
+ return -1;
+
+ rc = ExpLOBinterfacePerformGC(lobGlobL,tgtLobName, (void *)dcArray,
numEntries,hdfsServer,hdfsPort,lobLoc,lobMaxChunkMemSize);
+
+ return rc;
+}
+
+Int32 ExpLOBoper::restoreLobDataFile(void *lobGlob, char *lobName, void
*lobHeap, char *hdfsServer, Int32 hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+return -1;
+
+ rc =
ExpLOBinterfaceRestoreLobDataFile(lobGlobL,hdfsServer,hdfsPort,lobLoc,lobName);
+ return rc;
+}
+
+Int32 ExpLOBoper::purgeBackupLobDataFile(void *lobGlob,char *lobName, void
*lobHeap, char * hdfsServer, Int32 hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+ rc = initLOBglobal(lobGlobL, lobHeap);
--- End diff --
Same comment as in the prior method.
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185298#comment-15185298
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55393378
--- Diff: core/sql/exp/ExpLOB.cpp ---
@@ -186,13 +173,114 @@ Lng32 ExpLOBoper::createLOB(void * lobGlob, void *
lobHeap,
else
lobGlobL = lobGlob;
- rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,(char *)"default",lobMaxSize);
+ rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,hdfsServer,lobMaxSize, hdfsPort);
+
+ return rc;
+}
+void ExpLOBoper::calculateNewOffsets(ExLobInMemoryDescChunksEntry
*dcArray, Lng32 numEntries)
+{
+ Int32 i = 0;
+ //Check if there is a hole right up front for the first entry. If so
start compacting with the first entry.
+ if (dcArray[0].getCurrentOffset() != 0)
+{
+ dcArray[0].setNewOffset(0);
+ for (i = 1; i < numEntries; i++)
+{
+ dcArray[i].setNewOffset(dcArray[i-1].getNewOffset() +
dcArray[i-1].getChunkLen());
+}
+}
+ else
+//Look for the first unused section and start compacting from there.
+{
+ NABoolean done = FALSE;
+ i = 0;
+ Int32 j = 0;
+ while (i < numEntries && !done )
+{
+ if ((dcArray[i].getCurrentOffset()+dcArray[i].getChunkLen()) !=
+ dcArray[i+1].getCurrentOffset())
+{
+ j = i+1;
+ while (j < numEntries)
+{
+
dcArray[j].setNewOffset(dcArray[j-1].getNewOffset()+dcArray[j-1].getChunkLen());
+ j++;
+}
+ done = TRUE;
+}
+ i++;
+}
+}
+ return ;
+}
+
+Lng32 ExpLOBoper::compactLobDataFile(void
*lobGlob,ExLobInMemoryDescChunksEntry *dcArray,Int32 numEntries,char
*tgtLobName,Int64 lobMaxChunkMemSize, void *lobHeap, char *hdfsServer, Int32
hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ // Call ExeLOBinterface to create the LOB
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+
+ if (rc)
+ return -1;
+
+ rc = ExpLOBinterfacePerformGC(lobGlobL,tgtLobName, (void *)dcArray,
numEntries,hdfsServer,hdfsPort,lobLoc,lobMaxChunkMemSize);
+
+ return rc;
+}
+
+Int32 ExpLOBoper::restoreLobDataFile(void *lobGlob, char *lobName, void
*lobHeap, char *hdfsServer, Int32 hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+ rc = initLOBglobal(lobGlobL, lobHeap);
--- End diff --
Not sure I understand the point of the 'if'. In both cases, lobGlobL will
be equal to lobGlob when we call initLOBglobal. Why not just code "rc =
initLOBglobal(lobGlob, lobHeap)" once and be done with it?
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185293#comment-15185293
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55392981
--- Diff: core/sql/exp/ExpLOB.cpp ---
@@ -186,13 +173,114 @@ Lng32 ExpLOBoper::createLOB(void * lobGlob, void *
lobHeap,
else
lobGlobL = lobGlob;
- rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,(char *)"default",lobMaxSize);
+ rc = ExpLOBinterfaceCreate(lobGlobL, lobName, lobLoc,
Lob_HDFS_File,hdfsServer,lobMaxSize, hdfsPort);
+
+ return rc;
+}
+void ExpLOBoper::calculateNewOffsets(ExLobInMemoryDescChunksEntry
*dcArray, Lng32 numEntries)
+{
+ Int32 i = 0;
+ //Check if there is a hole right up front for the first entry. If so
start compacting with the first entry.
+ if (dcArray[0].getCurrentOffset() != 0)
+{
+ dcArray[0].setNewOffset(0);
+ for (i = 1; i < numEntries; i++)
+{
+ dcArray[i].setNewOffset(dcArray[i-1].getNewOffset() +
dcArray[i-1].getChunkLen());
+}
+}
+ else
+//Look for the first unused section and start compacting from there.
+{
+ NABoolean done = FALSE;
+ i = 0;
+ Int32 j = 0;
+ while (i < numEntries && !done )
+{
+ if ((dcArray[i].getCurrentOffset()+dcArray[i].getChunkLen()) !=
+ dcArray[i+1].getCurrentOffset())
+{
+ j = i+1;
+ while (j < numEntries)
+{
+
dcArray[j].setNewOffset(dcArray[j-1].getNewOffset()+dcArray[j-1].getChunkLen());
+ j++;
+}
+ done = TRUE;
+}
+ i++;
+}
+}
+ return ;
+}
+
+Lng32 ExpLOBoper::compactLobDataFile(void
*lobGlob,ExLobInMemoryDescChunksEntry *dcArray,Int32 numEntries,char
*tgtLobName,Int64 lobMaxChunkMemSize, void *lobHeap, char *hdfsServer, Int32
hdfsPort, char *lobLoc)
+{
+ Int32 rc = 0;
+ void * lobGlobL = NULL;
+ // Call ExeLOBinterface to create the LOB
+ if (lobGlob == NULL)
+{
+ rc = initLOBglobal(lobGlobL, lobHeap);
+ if (rc)
+ return -1;
+}
+ else
+lobGlobL = lobGlob;
+
+ if (rc)
--- End diff --
Harmless but unnecessary as this 'if' can never be true.
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185269#comment-15185269
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55391332
--- Diff: core/sql/cli/Cli.cpp ---
@@ -9581,15 +9531,14 @@ Lng32 SQLCLI_LOBcliInterface
if (cliRC < 0)
{
- cliInterface->executeImmediate("cqd pos restore;");
-
+
cliInterface->retrieveSQLDiagnostics(myDiags);
goto error_return;
}
// create lob descriptor chunks table salted
- str_sprintf(query, "create ghost table %s (descPartnKey
largeint not null, descSysKey largeint not null, chunkNum int not null,
chunkLen largeint not null, intParam largeint, stringParam varchar(400),
primary key(descPartnKey, descSysKey, chunkNum)) salt using 8 partitions",
+ str_sprintf(query, "create ghost table %s (descPartnKey
largeint not null, descSysKey largeint not null, chunkNum int not null,
chunkLen largeint not null, dataOffset largeint, stringParam varchar(400),
primary key(descPartnKey, descSysKey, chunkNum)) salt using 8 partitions",
lobDescChunksName);
--- End diff --
Does renaming a column imply an upgrade step when this software is
installed?
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185265#comment-15185265
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55391151
--- Diff: core/sql/cli/Cli.cpp ---
@@ -9684,15 +9640,17 @@ Lng32 SQLCLI_LOBcliInterface
cliRC = cliInterface->executeImmediate(query);
currContext.resetSqlParserFlags(0x1);
-
+
if (cliRC < 0)
{
cliInterface->retrieveSQLDiagnostics(myDiags);
-
+ if (myDiags->containsError(-CAT_OBJECT_DOES_NOT_EXIST_IN_TRAFODION))
--- End diff --
What if there are other errors in myDiags along with
CAT_OBJECT_DOES_NOT_EXIST_IN_TRAFODION; is it OK to ignore them as well?
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[
https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183323#comment-15183323
]
ASF GitHub Bot commented on TRAFODION-1848:
---
Github user arvind-narain commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/363#discussion_r55239271
--- Diff: core/sql/cli/Cli.cpp ---
@@ -9671,7 +9622,12 @@ Lng32 SQLCLI_LOBcliInterface
if (cliRC < 0)
{
cliInterface->retrieveSQLDiagnostics(myDiags);
-
+cliInterface->retrieveSQLDiagnostics(myDiags);
--- End diff --
This got duplicated - could be removed.
> LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr
> process
> --
>
> Key: TRAFODION-1848
> URL: https://issues.apache.org/jira/browse/TRAFODION-1848
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-exe
>Affects Versions: any
>Reporter: Sandhya Sundaresan
>Assignee: Sandhya Sundaresan
> Fix For: 2.0-incubating
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Need support to compact lob data files after a certain threshold is reached .
> Deleting or updating a table with lob columns cn cause the lob datafiles to
> be left with unused holes. These need to be compacted regularly.
> TO simplify the flow of execution, the mxlobsrvr process functionality is
> being moved into the master executor. There is no need for a spearate process
> anymore since there isn't going to be a separate storage engine that may need
> to make calls to this process. The lob file processing can all be done in
> the master process itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[ https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179479#comment-15179479 ] ASF GitHub Bot commented on TRAFODION-1848: --- GitHub user sandhyasun opened a pull request: https://github.com/apache/incubator-trafodion/pull/363 Support for doing GC on LOB data files and removing functionality from mxlobsrvr process TRAFODION-1848 The following set of changes are being committed : 1. Moved all calls made in mxlobsrvr process into the master. Removed interprocess sned/recieve calls and made them function calls all done within master. Retianed the minimum infrastructure to mxlobsrvr but it's not started anymore during sqstart. 2. Add a new LOB call to perform GC. Made this a new CLI call so we could extend it later to be a utility that can be called offline. For now this CLI cll will be triggered if a certain threshold is met during an insert/update/append operation. This GC is done per lob column (i.e per lob data file). Each lob column's GC can be done independently and do not need to be done all at once. Dpeneds on how much space each lob column ins occupying. 3 . Removed all uses of hardcoded references ot lob location and hdfsServer and hdfs port. ALl these are set via CQDs and are passed in to needed Tdbs. 4. A new testcase has been added to executor/TEST130 to exercise the new code. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sandhyasun/incubator-trafodion lob_gc_work Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/363.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #363 > LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr > process > -- > > Key: TRAFODION-1848 > URL: https://issues.apache.org/jira/browse/TRAFODION-1848 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-exe >Affects Versions: any >Reporter: Sandhya Sundaresan >Assignee: Sandhya Sundaresan > Fix For: 2.0-incubating > > Original Estimate: 672h > Remaining Estimate: 672h > > Need support to compact lob data files after a certain threshold is reached . > Deleting or updating a table with lob columns cn cause the lob datafiles to > be left with unused holes. These need to be compacted regularly. > TO simplify the flow of execution, the mxlobsrvr process functionality is > being moved into the master executor. There is no need for a spearate process > anymore since there isn't going to be a separate storage engine that may need > to make calls to this process. The lob file processing can all be done in > the master process itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-1848) LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr process
[ https://issues.apache.org/jira/browse/TRAFODION-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179433#comment-15179433 ] Sandhya Sundaresan commented on TRAFODION-1848: --- Changes have been made to perform GC as part of an insert or update operation. As a further improvement we should probably add a utility operation that does GC in an offline mode. For now, GC kicks in during an insert/update or append, when the lob data file has reached a certain threshold. The threshold is controllable via a CQD. All functionality has moved into the master executor process including the new GC function. The infrastructure to launch and send messages to the mxlobsrvr process from the master is still retained but no message is actually sent to it now. We may use it in the future for other functions and may even move GC to that process. The mxlobsrvr executable will still be built but it will not be launched as part of process startup since it is not needed right now. As part of moving this functionality also refactored some of the code. > LOB infrastructure changes to support GC and get rid of unneeded mxlobsrvr > process > -- > > Key: TRAFODION-1848 > URL: https://issues.apache.org/jira/browse/TRAFODION-1848 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-exe >Affects Versions: any >Reporter: Sandhya Sundaresan >Assignee: Sandhya Sundaresan > Fix For: 2.0-incubating > > Original Estimate: 672h > Remaining Estimate: 672h > > Need support to compact lob data files after a certain threshold is reached . > Deleting or updating a table with lob columns cn cause the lob datafiles to > be left with unused holes. These need to be compacted regularly. > TO simplify the flow of execution, the mxlobsrvr process functionality is > being moved into the master executor. There is no need for a spearate process > anymore since there isn't going to be a separate storage engine that may need > to make calls to this process. The lob file processing can all be done in > the master process itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
