Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
Thanks again and sorry to bother you guys but I'm new to Github and don't
know what do do from here. Can you point me to the right place where I can
take the next step to put this patch on my server? I only know how to
untar the tarball I downloaded from the main ES page.
Thanks.
Tony
On Wednesday, August 27, 2014 1:35:06 PM UTC-4, tony@iqor.com wrote:
Kudos!
Tony
On Wednesday, August 27, 2014 1:16:11 PM UTC-4, Jörg Prante wrote:
All praise should go to the fantastic Elasticsearch team who did not
hesitate to test the fix immediately and replaced it with a better working
solution, since the lzf-compress software is having weaknesses regarding
threadsafety.
Jörg
On Wed, Aug 27, 2014 at 7:01 PM, Ivan Brusic iv...@brusic.com wrote:
Amazing job. Great work.
--
Ivan
On Tue, Aug 26, 2014 at 12:41 PM, joerg...@gmail.com joerg...@gmail.com
wrote:
I fixed the issue by setting the safe LZF encoder in LZFCompressor and
opened a pull request
https://github.com/elasticsearch/elasticsearch/pull/7466
Jörg
On Tue, Aug 26, 2014 at 8:17 PM, joerg...@gmail.com joerg...@gmail.com
wrote:
Still broken with lzf-compress 1.0.3
https://gist.github.com/jprante/d2d829b497db4963aea5
Jörg
On Tue, Aug 26, 2014 at 7:54 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
Thanks for the logstash mapping command. I can reproduce it now.
It's the LZF encoder that bails out at
org.elasticsearch.common.compress.lzf.impl.UnsafeChunkEncoderBE._getInt
which uses in turn sun.misc.Unsafe.getInt
I have created a gist of the JVM crash file at
https://gist.github.com/jprante/79f4b4c0b9fd83eb1c9b
There has been a fix in LZF lately
https://github.com/ning/compress/commit/db7f51bddc5b7beb47da77eeeab56882c650bff7
for version 1.0.3 which has been released recently.
I will build a snapshot ES version with LZF 1.0.3 and see if this
works...
Jörg
On Mon, Aug 25, 2014 at 11:30 PM, tony@iqor.com wrote:
I captured a WireShark trace of the interaction between ES and
Logstash 1.4.1. The error occurs even before my data is sent. Can you
try
to reproduce it on your testbed with this message I captured?
curl -XPUT http://amssc103-mgmt-app2:9200/_template/logstash -d @y
Contests of file 'y:
{ template : logstash-*, settings : {
index.refresh_interval : 5s }, mappings : {_default_ : {
_all : {enabled : true}, dynamic_templates : [ {
string_fields : { match : *,
match_mapping_type
: string, mapping : { type : string,
index
: analyzed, omit_norms : true, fields : {
raw : {type: string, index : not_analyzed, ignore_above :
256} } } } } ],
properties :
{ @version: { type: string, index: not_analyzed },
geoip : { type : object, dynamic:
true,
path: full, properties : {
location : { type : geo_point } } } }
}
}}
On Monday, August 25, 2014 3:53:18 PM UTC-4, tony@iqor.com
wrote:
I have no plugins installed (yet) and only changed
es.logger.level to DEBUG in logging.yml.
elasticsearch.yml:
cluster.name: es-AMS1Cluster
node.name: KYLIE1
node.rack: amssc2client02
path.data: /export/home/apontet/elasticsearch/data
path.work: /export/home/apontet/elasticsearch/work
path.logs: /export/home/apontet/elasticsearch/logs
network.host: = sanitized line; file contains
actual server IP
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [s1, s2, s3, s5 , s6,
s7] = Also sanitized
Thanks,
Tony
On Saturday, August 23, 2014 6:29:40 AM UTC-4, Jörg Prante wrote:
I tested a simple Hello World document on Elasticsearch 1.3.2
with Oracle JDK 1.7.0_17 64-bit Server VM, Sparc Solaris 10, default
settings.
No issues.
So I would like to know more about the settings in
elasticsearch.yml, the mappings, and the installed plugins.
Jörg
On Sat, Aug 23, 2014 at 11:25 AM, joerg...@gmail.com
joerg...@gmail.com wrote:
I have some Solaris 10 Sparc V440/V445 servers available and can
try to reproduce over the weekend.
Jörg
On Sat, Aug 23, 2014 at 4:37 AM, Robert Muir
rober...@elasticsearch.com wrote:
How big is it? Maybe i can have it anyway? I pulled two ancient
ultrasparcs out of my closet to try to debug your issue, but
unfortunately
they are a pita to work with (dead nvram battery on both, zeroed
mac
address, etc.) Id still love to get to the bottom of this.
On Aug 22, 2014 3:59 PM, tony@iqor.com wrote:
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the
process image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand
wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the
Core dump written line that
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
The easiest for me is to install fresh binaries but I'm not shy about
learning about Maven while I build it from source.
Thanks
Tony
On Friday, August 29, 2014 11:21:34 AM UTC-4, Jörg Prante wrote:
Do you want to build from source? Or do you want to install a fresh binary?
At jenkins.elasticsearch.org I can not find any snapshot builds but it
may be just me.
It would be a nice add-on to provide snapshot builds for users that
eagerly await bug fixes or take a ride on the bleeding edge before the next
release arrives, without release notes etc.
Jörg
On Fri, Aug 29, 2014 at 4:29 PM, tony@iqor.com javascript: wrote:
Thanks again and sorry to bother you guys but I'm new to Github and don't
know what do do from here. Can you point me to the right place where I can
take the next step to put this patch on my server? I only know how to
untar the tarball I downloaded from the main ES page.
Thanks.
Tony
On Wednesday, August 27, 2014 1:35:06 PM UTC-4, tony@iqor.com wrote:
Kudos!
Tony
On Wednesday, August 27, 2014 1:16:11 PM UTC-4, Jörg Prante wrote:
All praise should go to the fantastic Elasticsearch team who did not
hesitate to test the fix immediately and replaced it with a better working
solution, since the lzf-compress software is having weaknesses regarding
threadsafety.
Jörg
On Wed, Aug 27, 2014 at 7:01 PM, Ivan Brusic iv...@brusic.com wrote:
Amazing job. Great work.
--
Ivan
On Tue, Aug 26, 2014 at 12:41 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
I fixed the issue by setting the safe LZF encoder in LZFCompressor
and opened a pull request
https://github.com/elasticsearch/elasticsearch/pull/7466
Jörg
On Tue, Aug 26, 2014 at 8:17 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
Still broken with lzf-compress 1.0.3
https://gist.github.com/jprante/d2d829b497db4963aea5
Jörg
On Tue, Aug 26, 2014 at 7:54 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
Thanks for the logstash mapping command. I can reproduce it now.
It's the LZF encoder that bails out at org.elasticsearch.common.
compress.lzf.impl.UnsafeChunkEncoderBE._getInt
which uses in turn sun.misc.Unsafe.getInt
I have created a gist of the JVM crash file at
https://gist.github.com/jprante/79f4b4c0b9fd83eb1c9b
There has been a fix in LZF lately https://github.com/
ning/compress/commit/db7f51bddc5b7beb47da77eeeab56882c650bff7
for version 1.0.3 which has been released recently.
I will build a snapshot ES version with LZF 1.0.3 and see if this
works...
Jörg
On Mon, Aug 25, 2014 at 11:30 PM, tony@iqor.com wrote:
I captured a WireShark trace of the interaction between ES and
Logstash 1.4.1. The error occurs even before my data is sent. Can
you try
to reproduce it on your testbed with this message I captured?
curl -XPUT http://amssc103-mgmt-app2:9200/_template/logstash -d @y
Contests of file 'y:
{ template : logstash-*, settings : {
index.refresh_interval : 5s }, mappings : {_default_ :
{
_all : {enabled : true}, dynamic_templates : [ {
string_fields : { match : *,
match_mapping_type
: string, mapping : { type : string,
index
: analyzed, omit_norms : true, fields : {
raw : {type: string, index : not_analyzed, ignore_above
:
256} } } } } ],
properties :
{ @version: { type: string, index: not_analyzed },
geoip : { type : object, dynamic:
true,
path: full, properties : {
location : { type : geo_point } } } }
}
}}
On Monday, August 25, 2014 3:53:18 PM UTC-4, tony@iqor.com
wrote:
I have no plugins installed (yet) and only changed
es.logger.level to DEBUG in logging.yml.
elasticsearch.yml:
cluster.name: es-AMS1Cluster
node.name: KYLIE1
node.rack: amssc2client02
path.data: /export/home/apontet/elasticsearch/data
path.work: /export/home/apontet/elasticsearch/work
path.logs: /export/home/apontet/elasticsearch/logs
network.host: = sanitized line; file contains
actual server IP
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [s1, s2, s3, s5 ,
s6, s7] = Also sanitized
Thanks,
Tony
On Saturday, August 23, 2014 6:29:40 AM UTC-4, Jörg Prante wrote:
I tested a simple Hello World document on Elasticsearch 1.3.2
with Oracle JDK 1.7.0_17 64-bit Server VM, Sparc Solaris 10,
default
settings.
No issues.
So I would like to know more about the settings in
elasticsearch.yml, the mappings, and the installed plugins.
Jörg
On Sat, Aug 23, 2014 at 11:25 AM, joerg...@gmail.com
joerg...@gmail.com wrote:
I have some Solaris 10 Sparc V440/V445 servers available and
can try to reproduce over the weekend.
Jörg
On Sat, Aug 23, 2014 at
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
Thank you very much.
Tony
On Friday, August 29, 2014 3:27:33 PM UTC-4, Jörg Prante wrote:
Quick guide:
- install Java 7 (or Java 8), Apache Maven, and git, also ensure internet
connection to the Maven central repo
- clone 1.3 branch only (you could also clone the whole repo and switch to
the branch): git clone https://github.com/elasticsearch/elasticsearch.git
--branch 1.3 --single-branch es-1.3
- enter folder es-1.3
- start build: mvn -DskipTests clean install
- wait a few minutes while Maven loads all dependent artifacts and
compiles ~3000 source files
The result will be a complete build of all binaries. In the 'target'
folder, after the Build complete message of Maven, you will see a file
elasticsearch-VERSION.jar
VERSION is something like 1.3.3-SNAPSHOT. You can copy this file into
your existing Elasticsearch 1.3.x installation lib folder. Do not forget
to adjust bin/elasticsearch.in.sh to point to the new
elasticsearch-VERSION.jar file in the classpath configuration (at the top
lines). This must be the first jar on the classpath so it can patch Lucene
jars.
If you have already data in the existing Elasticsearch I recommend to
backup everything before starting the new snapshot build - no guarantees,
use at your own risk.
Jörg
On Fri, Aug 29, 2014 at 7:36 PM, tony@iqor.com javascript: wrote:
The easiest for me is to install fresh binaries but I'm not shy about
learning about Maven while I build it from source.
Thanks
Tony
On Friday, August 29, 2014 11:21:34 AM UTC-4, Jörg Prante wrote:
Do you want to build from source? Or do you want to install a fresh
binary?
At jenkins.elasticsearch.org I can not find any snapshot builds but it
may be just me.
It would be a nice add-on to provide snapshot builds for users that
eagerly await bug fixes or take a ride on the bleeding edge before the next
release arrives, without release notes etc.
Jörg
On Fri, Aug 29, 2014 at 4:29 PM, tony@iqor.com wrote:
Thanks again and sorry to bother you guys but I'm new to Github and
don't know what do do from here. Can you point me to the right place
where
I can take the next step to put this patch on my server? I only know how
to untar the tarball I downloaded from the main ES page.
Thanks.
Tony
On Wednesday, August 27, 2014 1:35:06 PM UTC-4, tony@iqor.com
wrote:
Kudos!
Tony
On Wednesday, August 27, 2014 1:16:11 PM UTC-4, Jörg Prante wrote:
All praise should go to the fantastic Elasticsearch team who did not
hesitate to test the fix immediately and replaced it with a better
working
solution, since the lzf-compress software is having weaknesses regarding
threadsafety.
Jörg
On Wed, Aug 27, 2014 at 7:01 PM, Ivan Brusic iv...@brusic.com
wrote:
Amazing job. Great work.
--
Ivan
On Tue, Aug 26, 2014 at 12:41 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
I fixed the issue by setting the safe LZF encoder in LZFCompressor
and opened a pull request
https://github.com/elasticsearch/elasticsearch/pull/7466
Jörg
On Tue, Aug 26, 2014 at 8:17 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
Still broken with lzf-compress 1.0.3
https://gist.github.com/jprante/d2d829b497db4963aea5
Jörg
On Tue, Aug 26, 2014 at 7:54 PM, joerg...@gmail.com
joerg...@gmail.com wrote:
Thanks for the logstash mapping command. I can reproduce it now.
It's the LZF encoder that bails out at org.elasticsearch.common.
compress.lzf.impl.UnsafeChunkEncoderBE._getInt
which uses in turn sun.misc.Unsafe.getInt
I have created a gist of the JVM crash file at
https://gist.github.com/jprante/79f4b4c0b9fd83eb1c9b
There has been a fix in LZF lately https://github.com/ning
/compress/commit/db7f51bddc5b7beb47da77eeeab56882c650bff7
for version 1.0.3 which has been released recently.
I will build a snapshot ES version with LZF 1.0.3 and see if this
works...
Jörg
On Mon, Aug 25, 2014 at 11:30 PM, tony@iqor.com wrote:
I captured a WireShark trace of the interaction between ES and
Logstash 1.4.1. The error occurs even before my data is sent. Can
you try
to reproduce it on your testbed with this message I captured?
curl -XPUT http://amssc103-mgmt-app2:9200/_template/logstash -d
@y
Contests of file 'y:
{ template : logstash-*, settings : {
index.refresh_interval : 5s }, mappings : {_default_
: {
_all : {enabled : true}, dynamic_templates : [ {
string_fields : { match : *,
match_mapping_type
: string, mapping : { type : string,
index
: analyzed, omit_norms : true, fields : {
raw : {type: string, index : not_analyzed,
ignore_above :
256} } } } } ],
properties :
{ @version: { type: string, index: not_analyzed
},
geoip : { type : object, dynamic:
true,
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
Kudos!
Tony
On Wednesday, August 27, 2014 1:16:11 PM UTC-4, Jörg Prante wrote:
All praise should go to the fantastic Elasticsearch team who did not
hesitate to test the fix immediately and replaced it with a better working
solution, since the lzf-compress software is having weaknesses regarding
threadsafety.
Jörg
On Wed, Aug 27, 2014 at 7:01 PM, Ivan Brusic iv...@brusic.com
javascript: wrote:
Amazing job. Great work.
--
Ivan
On Tue, Aug 26, 2014 at 12:41 PM, joerg...@gmail.com javascript:
joerg...@gmail.com javascript: wrote:
I fixed the issue by setting the safe LZF encoder in LZFCompressor and
opened a pull request
https://github.com/elasticsearch/elasticsearch/pull/7466
Jörg
On Tue, Aug 26, 2014 at 8:17 PM, joerg...@gmail.com javascript:
joerg...@gmail.com javascript: wrote:
Still broken with lzf-compress 1.0.3
https://gist.github.com/jprante/d2d829b497db4963aea5
Jörg
On Tue, Aug 26, 2014 at 7:54 PM, joerg...@gmail.com javascript:
joerg...@gmail.com javascript: wrote:
Thanks for the logstash mapping command. I can reproduce it now.
It's the LZF encoder that bails out at
org.elasticsearch.common.compress.lzf.impl.UnsafeChunkEncoderBE._getInt
which uses in turn sun.misc.Unsafe.getInt
I have created a gist of the JVM crash file at
https://gist.github.com/jprante/79f4b4c0b9fd83eb1c9b
There has been a fix in LZF lately
https://github.com/ning/compress/commit/db7f51bddc5b7beb47da77eeeab56882c650bff7
for version 1.0.3 which has been released recently.
I will build a snapshot ES version with LZF 1.0.3 and see if this
works...
Jörg
On Mon, Aug 25, 2014 at 11:30 PM, tony@iqor.com javascript:
wrote:
I captured a WireShark trace of the interaction between ES and
Logstash 1.4.1. The error occurs even before my data is sent. Can you
try
to reproduce it on your testbed with this message I captured?
curl -XPUT http://amssc103-mgmt-app2:9200/_template/logstash -d @y
Contests of file 'y:
{ template : logstash-*, settings : {
index.refresh_interval : 5s }, mappings : {_default_ : {
_all : {enabled : true}, dynamic_templates : [ {
string_fields : { match : *,
match_mapping_type
: string, mapping : { type : string,
index
: analyzed, omit_norms : true, fields : {
raw : {type: string, index : not_analyzed, ignore_above :
256} } } } } ], properties
:
{ @version: { type: string, index: not_analyzed },
geoip : { type : object, dynamic: true,
path: full, properties : {
location : { type : geo_point } } } }
}
}}
On Monday, August 25, 2014 3:53:18 PM UTC-4, tony@iqor.com wrote:
I have no plugins installed (yet) and only changed es.logger.level
to DEBUG in logging.yml.
elasticsearch.yml:
cluster.name: es-AMS1Cluster
node.name: KYLIE1
node.rack: amssc2client02
path.data: /export/home/apontet/elasticsearch/data
path.work: /export/home/apontet/elasticsearch/work
path.logs: /export/home/apontet/elasticsearch/logs
network.host: = sanitized line; file contains
actual server IP
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [s1, s2, s3, s5 , s6,
s7] = Also sanitized
Thanks,
Tony
On Saturday, August 23, 2014 6:29:40 AM UTC-4, Jörg Prante wrote:
I tested a simple Hello World document on Elasticsearch 1.3.2
with Oracle JDK 1.7.0_17 64-bit Server VM, Sparc Solaris 10, default
settings.
No issues.
So I would like to know more about the settings in
elasticsearch.yml, the mappings, and the installed plugins.
Jörg
On Sat, Aug 23, 2014 at 11:25 AM, joerg...@gmail.com
joerg...@gmail.com wrote:
I have some Solaris 10 Sparc V440/V445 servers available and can
try to reproduce over the weekend.
Jörg
On Sat, Aug 23, 2014 at 4:37 AM, Robert Muir
rober...@elasticsearch.com wrote:
How big is it? Maybe i can have it anyway? I pulled two ancient
ultrasparcs out of my closet to try to debug your issue, but
unfortunately
they are a pita to work with (dead nvram battery on both, zeroed mac
address, etc.) Id still love to get to the bottom of this.
On Aug 22, 2014 3:59 PM, tony@iqor.com wrote:
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the
process image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand
wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the
Core dump written line that you pasted)
On Thu, Aug 21, 2014 at 7:53 PM, tony@iqor.com wrote:
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server
to scale out of small x86 machine. I get a similar exception
running ES
with JAVA_OPTS=-d64. When Logstash 1.4.1 sends the
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
It's as big as my ES_HEAP_SIZE parameter, 30g.
Tony
On Friday, August 22, 2014 10:37:39 PM UTC-4, Robert Muir wrote:
How big is it? Maybe i can have it anyway? I pulled two ancient
ultrasparcs out of my closet to try to debug your issue, but unfortunately
they are a pita to work with (dead nvram battery on both, zeroed mac
address, etc.) Id still love to get to the bottom of this.
On Aug 22, 2014 3:59 PM, tony@iqor.com javascript: wrote:
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the process
image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the Core dump
written line that you pasted)
On Thu, Aug 21, 2014 at 7:53 PM, tony@iqor.com wrote:
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server to scale
out of small x86 machine. I get a similar exception running ES with
JAVA_OPTS=-d64. When Logstash 1.4.1 sends the first message I get the
error below on the ES process:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7a9a3d8c, pid=14473, tid=209
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xba3d8c] Unsafe_GetInt+0x158
#
# Core dump written. Default location:
/export/home/elasticsearch/elasticsearch-1.3.2/core
or core.14473
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--- T H R E A D ---
Current thread (0x000107078000): JavaThread
elasticsearch[KYLIE1][http_server_worker][T#17]{New I/O worker #147}
daemon [_thread_in_vm, id=209, stack(0x5b80,
0x5b84)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN),
si_addr=0x000709cc09e7
I can run ES using 32bit java but have to shrink ES_HEAPS_SIZE more
than I want to. Any assistance would be appreciated.
Regards,
Tony
On Tuesday, July 22, 2014 5:43:28 AM UTC-4, David Roberts wrote:
Hello,
After upgrading from Elasticsearch 1.0.1 to 1.2.2 I'm getting JVM core
dumps on Solaris 10 on SPARC.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7e452d78, pid=15483, tid=263
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build
1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xc52d78] Unsafe_GetLong+0x158
I'm pretty sure the problem here is that Elasticsearch is making
increasing use of unsafe functions in Java, presumably to speed things
up, and some CPUs are more picky than others about memory alignment. In
particular, x86 will tolerate misaligned memory access whereas SPARC
won't.
Somebody has tried to report this to Oracle in the past and
(understandably) Oracle has said that if you're going to use unsafe
functions you need to understand what you're doing:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021574
A quick grep through the code of the two versions of Elasticsearch
shows that the new use of unsafe memory access functions is in the
BytesReference, MurmurHash3 and HyperLogLogPlusPlus classes:
bash-3.2$ git checkout v1.0.1
Checking out files: 100% (2904/2904), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/util/UnsafeUtils.java:public
enum UnsafeUtils {
./src/main/java/org/elasticsearch/search/aggregations/bucket/
BytesRefHash.java:if (id == -1L ||
UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/search/aggregations/bucket/
BytesRefHash.java:} else if (UnsafeUtils.equals(key,
get(curId, spare))) {
./src/test/java/org/elasticsearch/benchmark/common/util/Byte
sRefComparisonsBenchmark.java:import org.elasticsearch.common.util.
UnsafeUtils;
./src/test/java/org/elasticsearch/benchmark/common/util/Byte
sRefComparisonsBenchmark.java:return
UnsafeUtils.equals(b1, b2);
bash-3.2$ git checkout v1.2.2
Checking out files: 100% (2220/2220), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/bytes/BytesReference.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/bytes/BytesReferenc
e.java:return UnsafeUtils.equals(a.array(),
a.arrayOffset(), b.array(), b.arrayOffset(), a.length());
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:
return UnsafeUtils.readLongLE(key, blockOffset);
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
I have no plugins installed (yet) and only changed es.logger.level to
DEBUG in logging.yml.
elasticsearch.yml:
cluster.name: es-AMS1Cluster
node.name: KYLIE1
node.rack: amssc2client02
path.data: /export/home/apontet/elasticsearch/data
path.work: /export/home/apontet/elasticsearch/work
path.logs: /export/home/apontet/elasticsearch/logs
network.host: = sanitized line; file contains actual
server IP
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [s1, s2, s3, s5 , s6, s7]
= Also sanitized
Thanks,
Tony
On Saturday, August 23, 2014 6:29:40 AM UTC-4, Jörg Prante wrote:
I tested a simple Hello World document on Elasticsearch 1.3.2 with
Oracle JDK 1.7.0_17 64-bit Server VM, Sparc Solaris 10, default settings.
No issues.
So I would like to know more about the settings in elasticsearch.yml, the
mappings, and the installed plugins.
Jörg
On Sat, Aug 23, 2014 at 11:25 AM, joerg...@gmail.com javascript:
joerg...@gmail.com javascript: wrote:
I have some Solaris 10 Sparc V440/V445 servers available and can try to
reproduce over the weekend.
Jörg
On Sat, Aug 23, 2014 at 4:37 AM, Robert Muir rober...@elasticsearch.com
javascript: wrote:
How big is it? Maybe i can have it anyway? I pulled two ancient
ultrasparcs out of my closet to try to debug your issue, but unfortunately
they are a pita to work with (dead nvram battery on both, zeroed mac
address, etc.) Id still love to get to the bottom of this.
On Aug 22, 2014 3:59 PM, tony@iqor.com javascript: wrote:
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the process
image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the Core
dump written line that you pasted)
On Thu, Aug 21, 2014 at 7:53 PM, tony@iqor.com wrote:
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server to
scale out of small x86 machine. I get a similar exception running ES
with
JAVA_OPTS=-d64. When Logstash 1.4.1 sends the first message I get the
error below on the ES process:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7a9a3d8c, pid=14473, tid=209
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xba3d8c] Unsafe_GetInt+0x158
#
# Core dump written. Default location:
/export/home/elasticsearch/elasticsearch-1.3.2/core
or core.14473
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--- T H R E A D ---
Current thread (0x000107078000): JavaThread
elasticsearch[KYLIE1][http_server_worker][T#17]{New I/O worker
#147} daemon [_thread_in_vm, id=209, stack(0x5b80,
0x5b84)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN),
si_addr=0x000709cc09e7
I can run ES using 32bit java but have to shrink ES_HEAPS_SIZE more
than I want to. Any assistance would be appreciated.
Regards,
Tony
On Tuesday, July 22, 2014 5:43:28 AM UTC-4, David Roberts wrote:
Hello,
After upgrading from Elasticsearch 1.0.1 to 1.2.2 I'm getting JVM
core dumps on Solaris 10 on SPARC.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7e452d78, pid=15483, tid=263
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build
1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xc52d78] Unsafe_GetLong+0x158
I'm pretty sure the problem here is that Elasticsearch is making
increasing use of unsafe functions in Java, presumably to speed
things
up, and some CPUs are more picky than others about memory alignment.
In
particular, x86 will tolerate misaligned memory access whereas SPARC
won't.
Somebody has tried to report this to Oracle in the past and
(understandably) Oracle has said that if you're going to use unsafe
functions you need to understand what you're doing:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021574
A quick grep through the code of the two versions of Elasticsearch
shows that the new use of unsafe memory access functions is in the
BytesReference, MurmurHash3 and HyperLogLogPlusPlus classes:
bash-3.2$ git checkout v1.0.1
Checking out files: 100% (2904/2904), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/util/UnsafeUtils.java:public
enum UnsafeUtils {
./src/main/java/org/elasticsearch/search/aggregations/bucket/
BytesRefHash.java:if (id == -1L ||
UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/search/aggregations/bucket/
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
I was able to trim the heap size and, consequently, the core file down to
about 530m.
Tony
On Monday, August 25, 2014 3:41:14 PM UTC-4, tony@iqor.com wrote:
It's as big as my ES_HEAP_SIZE parameter, 30g.
Tony
On Friday, August 22, 2014 10:37:39 PM UTC-4, Robert Muir wrote:
How big is it? Maybe i can have it anyway? I pulled two ancient
ultrasparcs out of my closet to try to debug your issue, but unfortunately
they are a pita to work with (dead nvram battery on both, zeroed mac
address, etc.) Id still love to get to the bottom of this.
On Aug 22, 2014 3:59 PM, tony@iqor.com wrote:
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the process
image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the Core dump
written line that you pasted)
On Thu, Aug 21, 2014 at 7:53 PM, tony@iqor.com wrote:
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server to scale
out of small x86 machine. I get a similar exception running ES with
JAVA_OPTS=-d64. When Logstash 1.4.1 sends the first message I get the
error below on the ES process:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7a9a3d8c, pid=14473, tid=209
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xba3d8c] Unsafe_GetInt+0x158
#
# Core dump written. Default location:
/export/home/elasticsearch/elasticsearch-1.3.2/core
or core.14473
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--- T H R E A D ---
Current thread (0x000107078000): JavaThread
elasticsearch[KYLIE1][http_server_worker][T#17]{New I/O worker
#147} daemon [_thread_in_vm, id=209, stack(0x5b80,
0x5b84)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN),
si_addr=0x000709cc09e7
I can run ES using 32bit java but have to shrink ES_HEAPS_SIZE more
than I want to. Any assistance would be appreciated.
Regards,
Tony
On Tuesday, July 22, 2014 5:43:28 AM UTC-4, David Roberts wrote:
Hello,
After upgrading from Elasticsearch 1.0.1 to 1.2.2 I'm getting JVM
core dumps on Solaris 10 on SPARC.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7e452d78, pid=15483, tid=263
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build
1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xc52d78] Unsafe_GetLong+0x158
I'm pretty sure the problem here is that Elasticsearch is making
increasing use of unsafe functions in Java, presumably to speed things
up, and some CPUs are more picky than others about memory alignment. In
particular, x86 will tolerate misaligned memory access whereas SPARC
won't.
Somebody has tried to report this to Oracle in the past and
(understandably) Oracle has said that if you're going to use unsafe
functions you need to understand what you're doing:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021574
A quick grep through the code of the two versions of Elasticsearch
shows that the new use of unsafe memory access functions is in the
BytesReference, MurmurHash3 and HyperLogLogPlusPlus classes:
bash-3.2$ git checkout v1.0.1
Checking out files: 100% (2904/2904), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/util/UnsafeUtils.java:public
enum UnsafeUtils {
./src/main/java/org/elasticsearch/search/aggregations/bucket/
BytesRefHash.java:if (id == -1L ||
UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/search/aggregations/bucket/
BytesRefHash.java:} else if (UnsafeUtils.equals(key,
get(curId, spare))) {
./src/test/java/org/elasticsearch/benchmark/common/util/Byte
sRefComparisonsBenchmark.java:import org.elasticsearch.common.util.
UnsafeUtils;
./src/test/java/org/elasticsearch/benchmark/common/util/Byte
sRefComparisonsBenchmark.java:return
UnsafeUtils.equals(b1, b2);
bash-3.2$ git checkout v1.2.2
Checking out files: 100% (2220/2220), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/bytes/BytesReference.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/bytes/BytesReferenc
e.java:return UnsafeUtils.equals(a.array(),
a.arrayOffset(), b.array(), b.arrayOffset(), a.length());
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:import
org.elasticsearch.common.util.UnsafeUtils;
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
I captured a WireShark trace of the interaction between ES and Logstash
1.4.1. The error occurs even before my data is sent. Can you try to
reproduce it on your testbed with this message I captured?
curl -XPUT http://amssc103-mgmt-app2:9200/_template/logstash -d @y
Contests of file 'y:
{ template : logstash-*, settings : {index.refresh_interval :
5s }, mappings : {_default_ : { _all : {enabled :
true}, dynamic_templates : [ { string_fields : {
match : *, match_mapping_type : string,
mapping : { type : string, index : analyzed,
omit_norms : true, fields : { raw :
{type: string, index : not_analyzed, ignore_above : 256}
} } } } ], properties : {
@version: { type: string, index: not_analyzed }, geoip
: { type : object, dynamic: true,
path: full, properties : { location : {
type : geo_point } } } }} }}
On Monday, August 25, 2014 3:53:18 PM UTC-4, tony@iqor.com wrote:
I have no plugins installed (yet) and only changed es.logger.level to
DEBUG in logging.yml.
elasticsearch.yml:
cluster.name: es-AMS1Cluster
node.name: KYLIE1
node.rack: amssc2client02
path.data: /export/home/apontet/elasticsearch/data
path.work: /export/home/apontet/elasticsearch/work
path.logs: /export/home/apontet/elasticsearch/logs
network.host: = sanitized line; file contains actual
server IP
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [s1, s2, s3, s5 , s6, s7]
= Also sanitized
Thanks,
Tony
On Saturday, August 23, 2014 6:29:40 AM UTC-4, Jörg Prante wrote:
I tested a simple Hello World document on Elasticsearch 1.3.2 with
Oracle JDK 1.7.0_17 64-bit Server VM, Sparc Solaris 10, default settings.
No issues.
So I would like to know more about the settings in elasticsearch.yml, the
mappings, and the installed plugins.
Jörg
On Sat, Aug 23, 2014 at 11:25 AM, joerg...@gmail.com joerg...@gmail.com
wrote:
I have some Solaris 10 Sparc V440/V445 servers available and can try to
reproduce over the weekend.
Jörg
On Sat, Aug 23, 2014 at 4:37 AM, Robert Muir rober...@elasticsearch.com
wrote:
How big is it? Maybe i can have it anyway? I pulled two ancient
ultrasparcs out of my closet to try to debug your issue, but unfortunately
they are a pita to work with (dead nvram battery on both, zeroed mac
address, etc.) Id still love to get to the bottom of this.
On Aug 22, 2014 3:59 PM, tony@iqor.com wrote:
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the process
image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the Core
dump written line that you pasted)
On Thu, Aug 21, 2014 at 7:53 PM, tony@iqor.com wrote:
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server to
scale out of small x86 machine. I get a similar exception running ES
with
JAVA_OPTS=-d64. When Logstash 1.4.1 sends the first message I get the
error below on the ES process:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7a9a3d8c, pid=14473, tid=209
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xba3d8c] Unsafe_GetInt+0x158
#
# Core dump written. Default location:
/export/home/elasticsearch/elasticsearch-1.3.2/core
or core.14473
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--- T H R E A D ---
Current thread (0x000107078000): JavaThread
elasticsearch[KYLIE1][http_server_worker][T#17]{New I/O worker
#147} daemon [_thread_in_vm, id=209, stack(0x5b80,
0x5b84)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN),
si_addr=0x000709cc09e7
I can run ES using 32bit java but have to shrink ES_HEAPS_SIZE more
than I want to. Any assistance would be appreciated.
Regards,
Tony
On Tuesday, July 22, 2014 5:43:28 AM UTC-4, David Roberts wrote:
Hello,
After upgrading from Elasticsearch 1.0.1 to 1.2.2 I'm getting JVM
core dumps on Solaris 10 on SPARC.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7e452d78, pid=15483, tid=263
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build
1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xc52d78] Unsafe_GetLong+0x158
I'm pretty sure the problem here is that Elasticsearch is making
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
Hi Adrien,
It's a bunch of garbled binary data, basically a dump of the process image.
Tony
On Thursday, August 21, 2014 6:36:12 PM UTC-4, Adrien Grand wrote:
Hi Tony,
Do you have more information in the core dump file? (cf. the Core dump
written line that you pasted)
On Thu, Aug 21, 2014 at 7:53 PM, tony@iqor.com javascript: wrote:
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server to scale
out of small x86 machine. I get a similar exception running ES with
JAVA_OPTS=-d64. When Logstash 1.4.1 sends the first message I get the
error below on the ES process:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7a9a3d8c, pid=14473, tid=209
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xba3d8c] Unsafe_GetInt+0x158
#
# Core dump written. Default location:
/export/home/elasticsearch/elasticsearch-1.3.2/core or core.14473
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--- T H R E A D ---
Current thread (0x000107078000): JavaThread
elasticsearch[KYLIE1][http_server_worker][T#17]{New I/O worker #147}
daemon [_thread_in_vm, id=209, stack(0x5b80,0x5b84)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN),
si_addr=0x000709cc09e7
I can run ES using 32bit java but have to shrink ES_HEAPS_SIZE more than
I want to. Any assistance would be appreciated.
Regards,
Tony
On Tuesday, July 22, 2014 5:43:28 AM UTC-4, David Roberts wrote:
Hello,
After upgrading from Elasticsearch 1.0.1 to 1.2.2 I'm getting JVM core
dumps on Solaris 10 on SPARC.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7e452d78, pid=15483, tid=263
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build
1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xc52d78] Unsafe_GetLong+0x158
I'm pretty sure the problem here is that Elasticsearch is making
increasing use of unsafe functions in Java, presumably to speed things
up, and some CPUs are more picky than others about memory alignment. In
particular, x86 will tolerate misaligned memory access whereas SPARC won't.
Somebody has tried to report this to Oracle in the past and
(understandably) Oracle has said that if you're going to use unsafe
functions you need to understand what you're doing:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021574
A quick grep through the code of the two versions of Elasticsearch shows
that the new use of unsafe memory access functions is in the
BytesReference, MurmurHash3 and HyperLogLogPlusPlus classes:
bash-3.2$ git checkout v1.0.1
Checking out files: 100% (2904/2904), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/util/UnsafeUtils.java:public
enum UnsafeUtils {
./src/main/java/org/elasticsearch/search/aggregations/bucket/BytesRefHash.java:
if (id == -1L || UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/search/aggregations/bucket/BytesRefHash.java:
} else if (UnsafeUtils.equals(key, get(curId, spare))) {
./src/test/java/org/elasticsearch/benchmark/common/util/
BytesRefComparisonsBenchmark.java:import org.elasticsearch.common.util.
UnsafeUtils;
./src/test/java/org/elasticsearch/benchmark/common/util/
BytesRefComparisonsBenchmark.java:return
UnsafeUtils.equals(b1, b2);
bash-3.2$ git checkout v1.2.2
Checking out files: 100% (2220/2220), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/bytes/BytesReference.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/bytes/
BytesReference.java:return
UnsafeUtils.equals(a.array(), a.arrayOffset(), b.array(), b.arrayOffset(),
a.length());
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:
return UnsafeUtils.readLongLE(key, blockOffset);
./src/main/java/org/elasticsearch/common/hash/
MurmurHash3.java:long k1 = UnsafeUtils.readLongLE(key,
i);
./src/main/java/org/elasticsearch/common/hash/
MurmurHash3.java:long k2 = UnsafeUtils.readLongLE(key,
i + 8);
./src/main/java/org/elasticsearch/common/util/BytesRefHash.java:
if (id == -1L || UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/common/util/BytesRefHash.java:
} else if (UnsafeUtils.equals(key,
Re: JVM crash on 64 bit SPARC with Elasticsearch 1.2.2 due to unaligned memory access
Hello,
I installed ES 1.3.2 on a spare Solaris 11/ T4-4 SPARC server to scale out
of small x86 machine. I get a similar exception running ES with
JAVA_OPTS=-d64. When Logstash 1.4.1 sends the first message I get the
error below on the ES process:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7a9a3d8c, pid=14473, tid=209
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xba3d8c] Unsafe_GetInt+0x158
#
# Core dump written. Default location:
/export/home/elasticsearch/elasticsearch-1.3.2/core or core.14473
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
--- T H R E A D ---
Current thread (0x000107078000): JavaThread
elasticsearch[KYLIE1][http_server_worker][T#17]{New I/O worker #147}
daemon [_thread_in_vm, id=209, stack(0x5b80,0x5b84)]
siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN),
si_addr=0x000709cc09e7
I can run ES using 32bit java but have to shrink ES_HEAPS_SIZE more than I
want to. Any assistance would be appreciated.
Regards,
Tony
On Tuesday, July 22, 2014 5:43:28 AM UTC-4, David Roberts wrote:
Hello,
After upgrading from Elasticsearch 1.0.1 to 1.2.2 I'm getting JVM core
dumps on Solaris 10 on SPARC.
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0x7e452d78, pid=15483, tid=263
#
# JRE version: Java(TM) SE Runtime Environment (7.0_55-b13) (build
1.7.0_55-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.55-b03 mixed mode
solaris-sparc compressed oops)
# Problematic frame:
# V [libjvm.so+0xc52d78] Unsafe_GetLong+0x158
I'm pretty sure the problem here is that Elasticsearch is making
increasing use of unsafe functions in Java, presumably to speed things
up, and some CPUs are more picky than others about memory alignment. In
particular, x86 will tolerate misaligned memory access whereas SPARC won't.
Somebody has tried to report this to Oracle in the past and
(understandably) Oracle has said that if you're going to use unsafe
functions you need to understand what you're doing:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8021574
A quick grep through the code of the two versions of Elasticsearch shows
that the new use of unsafe memory access functions is in the
BytesReference, MurmurHash3 and HyperLogLogPlusPlus classes:
bash-3.2$ git checkout v1.0.1
Checking out files: 100% (2904/2904), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/util/UnsafeUtils.java:public enum
UnsafeUtils {
./src/main/java/org/elasticsearch/search/aggregations/bucket/BytesRefHash.java:
if (id == -1L || UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/search/aggregations/bucket/BytesRefHash.java:
} else if (UnsafeUtils.equals(key, get(curId, spare))) {
./src/test/java/org/elasticsearch/benchmark/common/util/BytesRefComparisonsBenchmark.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/test/java/org/elasticsearch/benchmark/common/util/BytesRefComparisonsBenchmark.java:
return UnsafeUtils.equals(b1, b2);
bash-3.2$ git checkout v1.2.2
Checking out files: 100% (2220/2220), done.
bash-3.2$ find . -name '*.java' | xargs grep UnsafeUtils
./src/main/java/org/elasticsearch/common/bytes/BytesReference.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/bytes/BytesReference.java:
return UnsafeUtils.equals(a.array(), a.arrayOffset(), b.array(),
b.arrayOffset(), a.length());
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:
return UnsafeUtils.readLongLE(key, blockOffset);
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:
long k1 = UnsafeUtils.readLongLE(key, i);
./src/main/java/org/elasticsearch/common/hash/MurmurHash3.java:
long k2 = UnsafeUtils.readLongLE(key, i + 8);
./src/main/java/org/elasticsearch/common/util/BytesRefHash.java:
if (id == -1L || UnsafeUtils.equals(key, get(id, spare))) {
./src/main/java/org/elasticsearch/common/util/BytesRefHash.java:
} else if (UnsafeUtils.equals(key, get(curId, spare))) {
./src/main/java/org/elasticsearch/common/util/UnsafeUtils.java:public enum
UnsafeUtils {
./src/main/java/org/elasticsearch/search/aggregations/metrics/cardinality/HyperLogLogPlusPlus.java:import
org.elasticsearch.common.util.UnsafeUtils;
./src/main/java/org/elasticsearch/search/aggregations/metrics/cardinality/HyperLogLogPlusPlus.java:
return
