[jira] [Updated] (CASSANDRA-8774) BulkOutputFormat never completes if streaming have errors
[ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-8774: - Priority: Minor (was: Major) > BulkOutputFormat never completes if streaming have errors > - > > Key: CASSANDRA-8774 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8774 > Project: Cassandra > Issue Type: Bug >Reporter: Erik Forsberg >Priority: Minor > Fix For: 2.1.x > > Attachments: > 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch > > > With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, > the hadoop task would fail. This doesn't seem to happen with 2.0.12. > I have a hadoop map task that use BulkoutputFormat. If one of the cassandra > nodes I'm writing to is down, I'm getting the following syslog output from > the map task: > {noformat} > 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded > the native-hadoop library > 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > Initializing JVM Metrics with processName=MAP, sessionId= > 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid > exited with exit code 0 > 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task: Using > ResourceCalculatorPlugin : > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e > 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing > split: > hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b1e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462 > 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: > Successfully loaded & initialized native-zlib library > 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got > brand-new decompressor > 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: > Unable to initialize MemoryMeter (jamm not specified as javaagent). This > means Cassandra will be unable to measure object sizes accurately and may > consequently OOM. > 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not > found. Native methods will be disabled. > 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: > Opening > /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_00_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1 > (1018 bytes) > 2015-02-10 10:54:20,713 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load > 2015-02-10 10:54:20,713 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with > /ipv6:prefix:1:441:0:0:0:7 > 2015-02-10 10:54:20,714 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with > /ipv6:prefix:1:441:0:0:0:8 > 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: > [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to > /ipv6:prefix:1:441:0:0:0:7 > 2015-02-10 10:54:20,730 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with > /ipv6:prefix:1:441:0:0:0:4 > 2015-02-10 10:54:20,750 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with > /ipv6:prefix:1:441:0:0:0:3 > 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: > [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to > /ipv6:prefix:1:441:0:0:0:8 > 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: > [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to > /ipv6:prefix:1:441:0:0:0:4 > 2015-02-10 10:54:20,770 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with > /ipv6:prefix:1:441:0:0:0:6 > 2015-02-10 10:54:20,778 INFO > org.apache.cassandra.streaming.StreamResultFuture: [Stream > #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with > /ipv6:prefix:1:441:0:0:0:5 > 2015-02-10 10:54:20,786 INFO
[jira] [Updated] (CASSANDRA-8774) BulkOutputFormat never completes if streaming have errors
[ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Forsberg updated CASSANDRA-8774: - Attachment: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch BulkOutputFormat never completes if streaming have errors - Key: CASSANDRA-8774 URL: https://issues.apache.org/jira/browse/CASSANDRA-8774 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Erik Forsberg Attachments: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, the hadoop task would fail. This doesn't seem to happen with 2.0.12. I have a hadoop map task that use BulkoutputFormat. If one of the cassandra nodes I'm writing to is down, I'm getting the following syslog output from the map task: {noformat} 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b1e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded initialized native-zlib library 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM. 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not found. Native methods will be disabled. 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: Opening /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_00_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1 (1018 bytes) 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:7 2015-02-10 10:54:20,714 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:8 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:7 2015-02-10 10:54:20,730 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:4 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:3 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:8 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:4 2015-02-10 10:54:20,770 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:6 2015-02-10 10:54:20,778 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:5 2015-02-10 10:54:20,786 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
[jira] [Updated] (CASSANDRA-8774) BulkOutputFormat never completes if streaming have errors
[ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8774: --- Fix Version/s: 2.0.13 BulkOutputFormat never completes if streaming have errors - Key: CASSANDRA-8774 URL: https://issues.apache.org/jira/browse/CASSANDRA-8774 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Erik Forsberg Fix For: 2.0.13 Attachments: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, the hadoop task would fail. This doesn't seem to happen with 2.0.12. I have a hadoop map task that use BulkoutputFormat. If one of the cassandra nodes I'm writing to is down, I'm getting the following syslog output from the map task: {noformat} 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b1e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded initialized native-zlib library 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM. 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not found. Native methods will be disabled. 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: Opening /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_00_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1 (1018 bytes) 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:7 2015-02-10 10:54:20,714 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:8 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:7 2015-02-10 10:54:20,730 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:4 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:3 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:8 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:4 2015-02-10 10:54:20,770 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:6 2015-02-10 10:54:20,778 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:5 2015-02-10 10:54:20,786 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to
[jira] [Updated] (CASSANDRA-8774) BulkOutputFormat never completes if streaming have errors
[ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8774: --- Reviewer: Yuki Morishita BulkOutputFormat never completes if streaming have errors - Key: CASSANDRA-8774 URL: https://issues.apache.org/jira/browse/CASSANDRA-8774 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Erik Forsberg Fix For: 2.0.13 Attachments: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, the hadoop task would fail. This doesn't seem to happen with 2.0.12. I have a hadoop map task that use BulkoutputFormat. If one of the cassandra nodes I'm writing to is down, I'm getting the following syslog output from the map task: {noformat} 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b1e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded initialized native-zlib library 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM. 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not found. Native methods will be disabled. 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: Opening /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_00_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1 (1018 bytes) 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:7 2015-02-10 10:54:20,714 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:8 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:7 2015-02-10 10:54:20,730 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:4 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:3 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:8 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to /ipv6:prefix:1:441:0:0:0:4 2015-02-10 10:54:20,770 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:6 2015-02-10 10:54:20,778 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:5 2015-02-10 10:54:20,786 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1] Starting streaming to