Hi again,
Few things are came into picture when I was debugging it.
DBStorage is setting Auto commit to False.
So when the batch is executed it will not be auto committed.
After executing batch OutputCommiter's method commitTask in DBStorage
(inline class' method) was called in which commit is written
if (ps != null) {
try {System.out.println("Executing Batch in commitTask");
ps.executeBatch();
con.commit();
ps.close();
con.close();
ps = null;
con = null;
} catch (SQLException e) {System.out.println("Exception in
commitTask");
log.error("ps.close", e);
throw new IOException("JDBC Error", e);
}
and this method is called by PigOutputCommiter
public void commitTask(TaskAttemptContext context) throws IOException {
if(HadoopShims.isMap(context.getTaskAttemptID())) {
for (Pair<OutputCommitter, POStore> mapCommitter :
mapOutputCommitters) {
if (mapCommitter.first!=null) {
TaskAttemptContext updatedContext =
setUpContext(context,
mapCommitter.second);
mapCommitter.first.commitTask(updatedContext);
}
}
} else {
for (Pair<OutputCommitter, POStore> reduceCommitter :
reduceOutputCommitters) {
if (reduceCommitter.first!=null) {
TaskAttemptContext updatedContext =
setUpContext(context,
reduceCommitter.second);
reduceCommitter.first.commitTask(updatedContext);
}
}
}
}
But when this commitTask is called its connection and preparedStatment
Objects become null... so it ll not commit so data is not available in
Database.....
But if you write only DBStorage without any other Store statement in script
it will work properly..
Any clues???
On Sun, May 26, 2013 at 11:40 PM, Hardik Shah <hardik.g...@gmail.com> wrote:
> DBStorage is not working with other storage in pig script. means DBStorage
> is not working with multiple storage statement.
>
> What I was trying for: 1) I was trying to Store one output using DBStorage
> And was trying to store same or different output using simple Store to file
> system 2) I also tried to store using DBStorage and using my custom store
> function
>
> But in both cases it not storing the data to database. If I comment out
> another storage than DBStorage is working properly.
>
> Even its not throwing any exception or error on reducer's machine..
>
> Can anyone point out the problem?
>
> DBStorage is not working with Simple Store to file system. Its only
> working if I put only DBStorage no other Store Statement..
>
> pv_by_industry = GROUP profile_view by viewee_industry_id
>
> pv_avg_by_industry = FOREACH pv_by_industry GENERATE
> group as viewee_industry_id, AVG(profie_view) AS average_pv;
>
> STORE pv_avg_by_industry INTO '/tmp/hardik';
>
> STORE pv_avg_by_industry into /tmp/hardik/db' INTO
> DBStorage('com.mysql.jdbc.Driver',
> 'dbc:mysql://hostname/dbname', 'user',
> 'pass',
> 'INSERT INTO table (viewee_industry_id,average_pv) VALUES(?,?)');
>
>
> --
> Regards,
> Hardik Shah.
>
--
Regards,
Hardik Shah.
